11institutetext: CERN, Geneva, Switzerland

# LHC Run-2 and Future Prospects

J. T. Boyd

###### Abstract

The lecture discusses both the current status of the LHC collider as well as
its future running scenarios. In addition a selection of the latest physics
results from ATLAS/CMS and LHCb is presented.

###### keywords:

LHC Run-2; EPSHEP2019.

##  1 The LHC Collider

The LHC is the highest energy particle collider in the world, situated in the
27 km long, former LEP tunnel at CERN. It is an extremely sophisticated
machine, using super conducting 8T dipole magnets to steer the high-energy
proton beams. The magnets are cooled to an operating temperature of 1.9K using
super-fluid liquid helium. Given the extreme energy of the beams, the LHC
needs a complex machine protection system, relying on a large number of beam
instrumentation devices to monitor the beam position and beam losses.

The two key parameters for a collider are the collision energy, and the
luminosity $L$ which is a measure of the number of collisions. The number of
events for a specific process ($N$) is given by $N=\sigma\times L$, where
$\sigma$ is the production cross-section for that process and $L$ is the
integrated luminosity.

The luminosity at a collider is given by the formula:
$L=n_{\rm{b}}N_{\rm{1}}N_{\rm{2}}F/4\pi\epsilon\beta^{\ast}$ and can be
increased by increasing the number of protons per bunch
($N_{\rm{1}},N_{\rm{2}}$), the number of colliding bunches ($n_{\rm{b}}$), or
reducing the transverse size of the beam at the collision point which can be
done by using a lower emittance ($\epsilon$) beam, or by squeezing the beam
more with the focussing magnets (reducing $\beta^{\ast}$). The crossing-angle
between the beams, needed to avoid parasitic collisions due to the short
distance between bunches, reduces the luminosity, and is encapsulated in the
geometric factor $F$ in the equation.

The main machine parameters for the LHC are shown in Table 1, for the design,
Run-1, Run-2, as well as the expectation for Run-3 and the High Luminosity
upgrade (HL-LHC). It can be seen that all of the design parameters have been
exceeded, except the collision energy, and the number of colliding bunches.
The LHC experts have continually improved the running scenario to increase the
luminosity, and during Run-2 the design luminosity of $10^{34}$ cm-2s-1, was
achieved and surpassed by a factor of two at the end of Run-2. As well as
improving the instantaneous luminosity, the availability of the machine was
dramatically improved during Run-2 which led to a large dataset for physics;
in 2016,2017,2018 running, the machine was providing physics collisions during
50% of the allocated physics time, which is very impressive for a super
conducting collider. An important parameter for the LHC experiments is the
pileup, which is determined by the luminosity per bunch, and is a measure of
the number of inelastic $pp$ interactions that occur per bunch crossing.
Higher pileup gives more luminosity (for a fixed number of bunches) but makes
physics analysis more difficult due to the signals in the detector from the
additional interactions.

The HL-LHC will operate from 2026 with the goal of delivering 3000 fb-1 in the
following ten years (an increase of a factor of ten compared to the expected
dataset at that time). In order to achieve this the injector needs to be
upgraded to provide higher intensity beam, the focussing magnets will be
replaced to be able to squeeze the beam more, and various components will be
upgraded to be able to cope with the increased radiation and stored energy.
The pileup in ATLAS and CMS will increase significantly (to 150 interactions
per bunch crossing) and the detectors will need large upgrades to be able to
make physics measurements at this large pileup, as well as to be able to cope
with the associated radiation.

Parameter | Design | Run-1 | Run-2 | Run-3 | HL-LHC
---|---|---|---|---|---
Energy [TeV] | 14 | 7/8 | 13 | 14 | 14
Bunch spacing [ns] | 25 | 50 | 25 | 25 | 25
Bunch Intensity [$10^{11}$ ppb] | 1.15 | 1.6 | 1.2 | up to 1.8 | 2.2
Number of bunches | 2800 | 1400 | 2500 | 2800 | 2800
Emittance [$\mu$m] | 3.5 | 2.2 | 2.2 | 2.5 | 2.5
$\beta^{*}$ [cm] | 55 | 80 | 30 $\to$ 25 | 30 $\to$ 25 | down to 15
Crossing angle [$\mu$rad] | 285 | - | 300 $\to$ 260 | 300 $\to$ 260 | TBD
Peak Luminosity [$10^{34}$ cm-2s-1] | 1.0 | 0.8 | 2.0 | 2.0 | 5.0
Peak pileup | 25 | 45 | 60 | 55 | 150
Table 1: Summary of main accelerator parameters for the LHC, showing the
design values, and those used during Run-1 and Run-2, as well as the expected
parameters for Run-3 and the HL-LHC.

##  2 Run-2 physics highlights and future prospects

###  2.1 The LHC detectors

ATLAS and CMS are the general-purpose detectors at the LHC with the same
physics goals. There are significant differences in the detector designs, but
despite these they have very similar physics performance.

The main differences in the detectors relate to the magnet design, ATLAS
utilizes a 2 T solenoid to provide the magnetic field to bend charged
particles in the central detector region, with three toroidal magnets (one
barrel toroid and two endcaps) to bend muons in the muon spectrometer. CMS
uses a single large solenoid with field of 3.8 T for both of these roles.
Following on from this the ATLAS calorimeters are placed outside the thin
solenoid, whereas in CMS the calorimeters are inside the solenoid.

###  2.2 The Run-2 dataset

During Run-2 ATLAS and CMS collected $\approx$140 fb-1 of data for physics
analysis. The cost of this large dataset was the significantly larger pileup
compared to the design value of 25, as can be seen in Figure 1, and it has
been one of the main challenges for the experiments to be able to efficiently
trigger, reconstruct the data, and carry out physics analyses with the Run-2
pileup. The experiments have put a huge effort into updating the trigger and
reconstruction algorithms to improve the performance at high pileup. Figure 1
shows examples of the achieved pileup robustness in terms of reconstructed
particles both for efficiency and resolution.

Figure 1: (left) The Run-2 pileup distribution. An example of the pileup
robustness of (middle) the reconstructed muon efficiency and (right) the
electron energy scale.

###  2.3 Higgs physics

The main modes for Higgs production at the LHC are (in order of decreasing
cross-section): gluon-fusion (ggF), Vector-Boson fusion (VBF), production in
association with a vector boson (VH) and production in association with a pair
of top-quarks (ttH). In VBF production the scattered quarks are likely to form
forward jets on the two sides of the detector, which can be used to tag such
events. The main decay modes for the Higgs boson are shown in Table 2.
Experimentally the modes with the best mass resolution are important, as this
allows to separate the signal from the background in a much more reliable way.
The $H\to\gamma\gamma$ and $H\to ZZ^{\ast}\to 4\ell$ ($\ell=e/\mu$) both have
excellent mass resolution of $\approx$1-2%, and these were the modes used for
the Higgs discovery in 2012, despite the fact they have very low branching
fractions (BF). Figure 2 shows the mass distributions for these two channels
for the full Run-2 dataset. For $H\to\gamma\gamma$ the signal to background
(S/B) is low, but the total number of selected Higgs events is a few thousand,
whereas for $H\to 4\ell$ the S/B is high, but the total number of signal
events is an order of magnitude less.

Poor mass resolution channels | Good mass resolution channels
---|---
Decay Mode | BF (%) | Decay Mode | BF (%)
$H\to b\overline{b}$ | 58.2 | $H\to ZZ^{\ast}$ | 2.6 (0.012 $e,\mu$)
$H\to WW^{\ast}$ | 21.4 (1.1 $e,\mu$) | $H\to\gamma\gamma$ | 0.23
$H\to gg$ | 8.2 | $H\to Z\gamma$ | 0.15 (0.008 $e,\mu$)
$H\to\tau^{+}\tau^{-}$ | 6.3 | $H\to\mu^{+}\mu^{-}$ | 0.02
$H\to c\overline{c}$ | 2.9 |  |
Table 2: Summary of Higgs decay modes (BFs and resolutions) for the 125 GeV
mass SM Higgs boson. For the good mass resolution channels involving a
Z-boson, the resolution is only good for leptonic decays of the Z.

Figure 2: Reconstructed Higgs candidate mass distributions in the
$H\to\gamma\gamma$ (left) and $H\to 4\ell$ (right) channels.

Table 3 shows the status of the main Higgs production and decay modes, in
terms of the significance of the measured signal. It shows that all of the
main production and decay modes have been established, although many of these
were only observed in the last year.

| $\gamma\gamma$ | $ZZ^{\ast}$ | $WW^{\ast}$ | $b\overline{b}$ | $c\overline{c}$ | $\tau^{+}\tau^{-}$ | $\mu^{+}\mu^{-}$ | Combined
---|---|---|---|---|---|---|---|---
ggF | Obs. | Obs. | Obs. | - | - | UL | UL | Obs.
VBF | UL | UL | UL | UL | - | Evid. | UL | Obs.
VH | UL | UL | UL | Obs. | UL | - | - | Obs.
ttH | Evid. | UL | Evid. | UL | - | Evid. | - | Obs.
Combined | Obs. | Obs. | Obs. | Obs. | UL | Obs. | UL | -
Table 3: Status of the measured significance for the main Higgs production and
decay modes. Here Obs./Evid. means the significance is at the level of an
observation/evidence, UL stands for ’Upper Limit’ and ’-’ implies this mode
has not been studied.

The Higgs coupling to fermions was established with the observation of the
$H\to\tau^{+}\tau^{-}$ decay. The analysis selects events with two $\tau$s
(either can decay hadronically or leptonically), and uses selections targeting
either VBF Higgs production or high- ggF Higgs production to reduce the
backgrounds. The main background is $Z\to\tau^{+}\tau^{-}$ which has the same
final state, with $\approx$1000$\times$ higher cross-section and with a
similar di-$\tau$ mass (the mass resolution is not sufficient to be able to
resolve the two processes). Figure 3 shows the di-$\tau$ mass distribution
from the Run-2 CMS analysis [1] where a tiny signal can be seen on top of the
large $Z\to\tau\tau$ background. The analysis measured the $H\to\tau\tau$ rate
to be compatible with the SM expectation with a precision of $\approx$30%,
corresponding to a 5.9$\sigma$ observation of the process. Searches for
$H\to\mu^{+}\mu^{-}$ have found no evidence of a signal (as expected in the SM
for the current dataset), which when combined with the $H\to\tau^{+}\tau^{-}$
result, demonstrates the Higgs couplings do not obey lepton flavour
conservation.

The Higgs coupling to quarks was established with the observation of $H\to
b\overline{b}$. Although this has the largest Higgs decay BF, it is
experimentally challenging due to the large background from QCD
$b\overline{b}$ production, and the poor di-$b$-jet mass resolution. In order
to reduce the background and to trigger on the events, the analysis targets VH
production where V is a $Z$ or $W$ boson decaying leptonically, so the final
state can have 0-leptons (but large missing transverse momentum (MET) from the
$Z\to\nu\overline{\nu}$ decay), 1-lepton, or 2-leptons, and selects two
$b$-jets with a mass close to the Higgs mass. As seen in Figure 3, which shows
the ATLAS analysis [2], the $VZ$, $Z\to b\overline{b}$ decay acts as an
important validation of the analysis. This has the same final state, with a
similar rate. The figure shows that $VZ$ is observed with the expected rate
(grey), and the $H\to b\overline{b}$ signal can be seen as a high mass
shoulder on the $Z$ peak. The analysis finds the expected SM rate with a
precision of $\approx$30%.

The Higgs is too light to decay to top quarks, so the top-Higgs (tH) coupling
can only be directly probed through ttH production. In the SM ggF production
is dominated by a top-quark in the ggF loop, and so the tH coupling can also
be extracted indirectly from ggF production rates. The direct and indirect
measurement of the coupling then allow to constrain possible new-particles
that could enter the ggF loop. Within the current precision the direct and
indirect measurements of the coupling are compatible.

Figure 3: (left) The di-tau mass distribution from the CMS
$H\to\tau^{+}\tau^{-}$ analysis; (right) The di-$b$-jet mass distribution from
the ATLAS $H\to b\overline{b}$ analysis.

With the increased luminosity at the HL-LHC the Higgs physics goals are:

  * •

Improve the precision on the Higgs couplings to the few-% level (where they
can be sensitive to BSM effects);

  * •

Establish the coupling to 2nd generation fermions through the
$H\to\mu^{+}\mu^{-}$ and $H\to c\overline{c}$ decays;

  * •

Improve the constraints on forbidden Higgs decays such as $H\to$ invisible and
lepton-flavour violating Higgs decays;

  * •

Make more precise differential measurements of Higgs production, in more
extreme regions of phase-space) which can be sensitive to New Physics;

  * •

Observe the very rare di-Higgs production process.

Studying di-Higgs production is needed to understand the Higgs self-coupling,
and to probe the Higgs potential term of the SM Lagrangian. However, it is
doubtful that this will be possible at the HL-LHC. Current projections [3] [4]
suggest that evidence for di-Higgs production can be achieved by combining the
ATLAS and CMS HL-LHC results.

###  2.4 Searches for Physics beyond the Standard Model

One of the primary goals of the LHC is to search for the direct production of
Beyond the Standard Model (BSM) physics. ATLAS and CMS have carried out a huge
number of searches, but to date no significant excess of events over the SM
expectation has been observed. A few example searches are discussed below.

A search for a new Gauge boson ($Z^{\prime}$) that is similar to the SM $Z$
boson but with much higher mass, looks for an excess of events in the di-
lepton mass spectra at high mass. Figure 4 shows the di-electron and di-muon
mass distributions from the ATLAS search [5]. No significant deviation from
the expected background (dominated by SM Drell-Yan production) is observed.
Examples signals are shown in the figures, which show that the mass resolution
is significantly better at high mass for electrons than for muons, as the
energy resolution improves for calorimeters, but deteriorates for tracking
detectors, at higher energy. The main experimental challenge for this search
is to have good efficiency and resolution for very high transverse momentum
leptons (up to 2 TeV).

Figure 4: The di-electron and di-muon mass distributions from the ATLAS
$Z^{\prime}$ search.

At the other end of the spectrum is a search for Higgsino production where
very low-momentum leptons are expected. The ATLAS search [6] uses leptons with
down to 3 GeV (muons) and 4.5 GeV (electrons) in order to improve the
sensitivity, and allows to exclude Higgsinos with masses up to 150 GeV for
certain mass splittings.

Searching for Dark Matter (DM) production in LHC collisions can be done by
taking advantage of initial-state-radiation which can be used to tag events
where DM particles are pair produced through an $s$-channel mediator particle
but escape the detector without interacting with it. This can lead to a
detector signature of a high- jet + MET. Figure 5 shows the MET spectrum for
such events from the CMS search [7], also showing the expected background,
dominated by $Z\to\nu\overline{\nu}$ \+ jets ($\approx$60%) and $W\to\ell\nu$
\+ jets (where the lepton is not reconstructed) ($\approx$30%). The signal has
a slightly harder MET-spectra than the background, but is much smaller than
the background, meaning the background needs to be controlled at the few-%
level to allow to have sensitivity. The background is estimated from data
control regions with $Z\to\ell^{+}\ell^{-}$ \+ jets, $W\to\ell\nu$ \+ jets and
$\gamma$ \+ jets but accurate theoretical predictions are needed on the ratio
of $Z$ \+ jets/$\gamma$ \+ jets and $Z$ \+ jets/$W$ \+ jets; in order to
achieve the needed precision nNLO electroweak corrections need to be taken
into account.

Figure 5: (left) The MET spectrum in the DM search; (middle) The di-jet mass
spectra in the mediator search; (right) The exclusion limit in the search for
the mediator showing results for both the high mass search, and the low mass
search which uses the Data Scouting technique.

As well as searching for the DM particle, we can also search for the mediator
which can be produced in the LHC collisions but decay back to the SM (for
example to two quarks) this could show up as a resonance in the di-jet mass
spectra. Figure 5 shows the di-jet mass distribution for such a CMS search
[8], showing a smoothly falling distribution with no sign of a resonance in
the range 1 TeV $\to$ 8 TeV in di-jet mass. The 1 TeV lower limit in the
probed mass range, comes from the trigger thresholds applied to the jets used
in the search. Going to lower -jets would increase the trigger rate leading to
a too high bandwidth when reading out the detector. In order to search for
possible resonances at lower mass a new technique called Data Scouting or
Trigger Level Analysis was developed, in which just the trigger level jets are
written out for certain triggers. These trigger level jets are much smaller
than the full event data (less than 5% of the size), and can therefore be read
out at a much higher rate without hitting bandwidth limitations. Thus lower
thresholds can be applied. This technique allows to set limits on di-jet mass
resonances down to lower masses as can be seen in Figure 5 (CMS analysis [8]).

###  2.5 Precise Standard Model Measurements

The LHC experiments carry out a large number of precise measurements of SM
processes, measuring cross-sections, masses and other SM parameters. Cross-
section measurements are normalized by the luminosity, which is measured by
dedicated luminosity detectors in the experiments that are calibrated by
dedicated van-der-Meer scans which are typically carried out each year. The
precision of the luminosity measurements in ATLAS/CMS for the Run-2 dataset is
an impressive $\approx$2.5%, which is far better than had thought to be
possible before LHC running.

An example of a very precise cross-section measurement is the $W$ and $Z$
inclusive production cross-section measurement from ATLAS [9] with the 2011 7
TeV dataset. The precision is limited by systematic uncertainties, and the
total experimental uncertainty is $\approx$0.5% dominated by uncertainties
related to the lepton reconstruction, the background (for the $W$) and
theoretical modeling uncertainties (for the $Z$). The luminosity uncertainty
is 1.8%, but this cancels in ratios such as $\sigma(W\to
e\nu)/\sigma(W\to\mu\nu)$ or $\sigma(W)/\sigma(Z)$ allowing very precise tests
of lepton flavour conservation, and parton distribution functions (PDFs).

The measurement of the $W$-boson mass by ATLAS [10] with a precision of 19 MeV
represents one of the most precise measurements at the LHC, and has a
precision equal to the best single-experiment measurement. The $W$ mass is a
fundamental parameter of the SM, and has important sensitivity in the
electroweak fit. The ATLAS analysis measures the mass using a template fits to
the transverse mass (formed from the lepton and the reconstructed hadronic
recoil), and to the lepton transverse momentum. A very precise knowledge of
experimental effects related to lepton reconstruction and the hadronic recoil
reconstruction is needed, where the later deteriorates significantly with
pileup. The current measurement utilizes the 2011 7 TeV data set which has an
average pileup of around 9. Theoretical uncertainties also play an important
role, in particular related to the modelling of the $W$-boson which is derived
from the measured $Z$-boson spectra, as well as from PDF uncertainties.
Utilizing low-pileup data taken in 2017 and 2018 at 13 TeV there is the
prospect of improving the precision of the measurement to the 10-15 MeV level.

Measurements of the top-quark mass are carried out in a number of different
channels. A recent example from CMS [11] utilizes the lepton+jets final state
to measure the mass using a kinematic fit (including the $W$-mass constraint
on the hadronic $W$ decay) to improve the resolution and to reduce the
fraction of incorrect assignments of jets to the two top-quarks. The dominant
systematic uncertainty is related to the jet energy scale which is constrained
in the fit. The final result of $172.25\pm 0.08$(stat.)$\pm 0.62$(syst.) GeV
is the most precise single measurement to date.

###  2.6 Flavour Physics

The $B_{S}\to\mu^{+}\mu^{-}$ rare decay is theoretically clean, and has a
large sensitivity to many new physics models (for example MSSM scenarios with
large Tan $\beta$). Because of this there is a long history of searches for
this decay starting over 30 years ago. Sensitivity to the SM branching ratio
of $(3.3\pm 0.3)\times 10^{-9}$ was reached with a combination between LHCb
and CMS [12]. Despite a much smaller dataset, LHCb has the best sensitivity
due to the excellent track resolution, as well as an optimized trigger for low
physics; CMS has better sensitivity than ATLAS due to the higher magnetic
field in the inner tracker which gives a better mass resolution. Current
measurements from all three experiments are consistent with the SM estimate
with an uncertainty of between 20 - 30 %.

LHCb searches for lepton flavour violation in $B$ meson decays by measuring
the ratio $R_{K^{(*)}}\equiv{\rm BF}(B\to K^{(*)}\mu^{+}\mu^{-})/{\rm BF}(B\to
K^{(*)}e^{+}e^{-})$. In the SM this is precisely predicted and is close to
unity, modulo phase-space effects. An experimental complication is that due to
bremsstrahlung the mass resolution is much worse in the di-electron channel
than the di-muon channel (as can be seen in Figure 6), but this is corrected
for by normalizing by the measured ratio BF$(B\to
K^{(*)}J/\psi(\mu^{+}\mu^{-}))$ / BF$(B\to K^{(*)}J/\psi(e^{+}e^{-}))$. For
$R_{K}$, the reconstructed $B$ meson mass distribution is shown in Figure 6 in
both the $\mu^{+}\mu^{-}$ and $e^{+}e^{-}$ channels. The measured values from
the LHCb measurements [13] [14] are shown in Table 4, and for $R_{K^{*}}$ in
Figure 6, and show that the three measurements are between 2 and 2.5 $\sigma$
lower than the SM prediction. This is currently one of the most intriguing
anomalies observed at the LHC, with many theoretical models proposed to
explain the results. More data and measurements from Belle-2 should shed
further light onto the situation.

Figure 6: (left/middle) The reconstructed $B$ meson mass in the $R_{K}$ analysis for the $\mu^{+}\mu^{-}$ / $e^{+}e^{-}$ channels; (right) The measured $R_{K^{*}}$ values in two bins of $q^{2}$ (the di-lepton mass) compared with various theoretical predictions. Measurement | Dataset | Measured Value | Compatibility with SM
---|---|---|---
$R_{K}$ | Run 1 + Run 2 | $0.85^{+0.06}_{-0.05}\pm 0.015$ | 2.5$\sigma$
$R_{K^{*}}$ low-q2 | Run 1 | $0.66^{+0.11}_{-0.07}\pm 0.03$ | 2.2$\sigma$
$R_{K^{*}}$ high-q2 | Run 1 | $0.69^{+0.11}_{-0.07}\pm 0.05$ | 2.4$\sigma$
Table 4: LHCb results on lepton flavour violation measurements $R_{K}$ and
$R_{K^{*}}$, where the latter is measured in two regions of $q^{2}$ (the di-
lepton mass).

##  3 Summary

The LHC machine and the experiments performed extremely well in Run-2. A large
and high-quality dataset was produced by the experiments leading to a huge
number of physics results. A leading challenge for the experiments was the
high pileup in the data, but they have coped very well with this situation.

The large dataset has allowed a more and more precise probing of the Higgs
boson, where all major production modes, and decay channels accessible at the
LHC have been established. A huge number of direct searches for BSM physics
have been carried out, with no significant excess of events over the SM
prediction observed, such that increasingly stringent exclusion limits have
been set on BSM model parameters. In addition, the experiments have been able
to make very precise measurements of cross-sections and SM parameters, as well
as measuring extremely rare processes, but again no discrepancy with the SM
expectations have been observed. An intriguing set of results from lepton-
flavour violation measurements by LHCb show a 2 - 2.5 standard deviation
discrepancy with the SM in a few channels and $q^{2}$-bins.

The increased dataset that will be produced with Run-3, and then with the HL-
LHC, along with the upgraded detector functionality, and innovations in
triggering, reconstruction and physics analysis will allow to probe further
the SM in the coming years.

## References

  * [1] A. M. Sirunyan et al. [CMS Collaboration], Phys. Lett. B 779 (2018) 283 doi:10.1016/j.physletb.2018.02.004 [arXiv:1708.00373 [hep-ex]].
  * [2] M. Aaboud et al. [ATLAS Collaboration], Phys. Lett. B 786 (2018) 59 doi:10.1016/j.physletb.2018.09.013 [arXiv:1808.08238 [hep-ex]].
  * [3] CMS CollaborationCMS-PAS-FTR-18-011, https://cds.cern.ch/record/2647699
  * [4] ATLAS Collaboration, ATL-PHYS-PUB-2018-054, http://cdsweb.cern.ch/record/2652762
  * [5] G. Aad et al. [ATLAS Collaboration], Phys. Lett. B 796 (2019) 68 doi:10.1016/j.physletb.2019.07.016 [arXiv:1903.06248 [hep-ex]].
  * [6] G. Aad et al. [ATLAS Collaboration], arXiv:1911.12606 [hep-ex].
  * [7] A. M. Sirunyan et al. [CMS Collaboration], Phys. Rev. D 97 (2018) no.9, 092005 doi:10.1103/PhysRevD.97.092005 [arXiv:1712.02345 [hep-ex]].
  * [8] A. M. Sirunyan et al. [CMS Collaboration], JHEP 1808 (2018) 130 doi:10.1007/JHEP08(2018)130 [arXiv:1806.00843 [hep-ex]].
  * [9] M. Aaboud et al. [ATLAS Collaboration], Eur. Phys. J. C 77 (2017) no.6, 367 doi:10.1140/epjc/s10052-017-4911-9 [arXiv:1612.03016 [hep-ex]].
  * [10] M. Aaboud et al. [ATLAS Collaboration], Eur. Phys. J. C 78 (2018) no.2, 110 Erratum: [Eur. Phys. J. C 78 (2018) no.11, 898] doi:10.1140/epjc/s10052-018-6354-3, 10.1140/epjc/s10052-017-5475-4 [arXiv:1701.07240 [hep-ex]].
  * [11] A. M. Sirunyan et al. [CMS Collaboration], Eur. Phys. J. C 78 (2018) no.11, 891 doi:10.1140/epjc/s10052-018-6332-9 [arXiv:1805.01428 [hep-ex]].
  * [12] V. Khachatryan et al. [CMS and LHCb Collaborations], Nature 522 (2015) 68 doi:10.1038/nature14474 [arXiv:1411.4413 [hep-ex]].
  * [13] R. Aaij et al. [LHCb Collaboration], JHEP 1708 (2017) 055 doi:10.1007/JHEP08(2017)055 [arXiv:1705.05802 [hep-ex]].
  * [14] R. Aaij et al. [LHCb Collaboration], Phys. Rev. Lett. 122 (2019) no.19, 191801 doi:10.1103/PhysRevLett.122.191801 [arXiv:1903.09252 [hep-ex]].


# MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset

Leonhard Hennig Philippe Thomas Sebastian Möller
German Research Center for Artificial Intelligence (DFKI)
Speech and Language Technology Lab
{leonhard.hennig, philippe.thomas<EMAIL_ADDRESS>
###### Abstract

Relation extraction (RE) is a fundamental task in information extraction,
whose extension to multilingual settings has been hindered by the lack of
supervised resources comparable in size to large English datasets such as
TACRED Zhang et al. (2017). To address this gap, we introduce the MultiTACRED
dataset, covering 12 typologically diverse languages from 9 language families,
which is created by machine-translating TACRED instances and automatically
projecting their entity annotations. We analyze translation and annotation
projection quality, identify error categories, and experimentally evaluate
fine-tuned pretrained mono- and multilingual language models in common
transfer learning scenarios. Our analyses show that machine translation is a
viable strategy to transfer RE instances, with native speakers judging more
than 83% of the translated instances to be linguistically and semantically
acceptable. We find monolingual RE model performance to be comparable to the
English original for many of the target languages, and that multilingual
models trained on a combination of English and target language data can
outperform their monolingual counterparts. However, we also observe a variety
of translation and annotation projection errors, both due to the MT systems
and linguistic features of the target languages, such as pronoun-dropping,
compounding and inflection, that degrade dataset quality and RE model
performance.

##  1 Introduction

Relation extraction (RE), defined as the task of identifying and classifying
semantic relationships between entities from text (cf. Figure 1), is a
fundamental task in information extraction Doddington et al. (2004). Extending
RE to multilingual settings has recently received increased interest Zou et
al. (2018); Nag et al. (2021); Chen et al. (2022c), both to address the urgent
need for more inclusive NLP systems that cover more languages than just
English Ruder et al. (2019); Hu et al. (2020), as well as to investigate
language-specific phenomena and challenges relevant to this task. The main
bottleneck for multilingual RE is the lack of supervised resources, comparable
in size to large English datasets Riedel et al. (2010); Zhang et al. (2017),
as annotation for new languages is very costly. Most of the few existing
multilingual RE datasets are distantly supervised Köksal and Özgür (2020);
Seganti et al. (2021); Bhartiya et al. (2022), and hence suffer from noisy
labels that may reduce the prediction quality of models Riedel et al. (2010);
Xie et al. (2021). Available fully-supervised datasets are small, and cover
either very few domain-specific relation types Arviv et al. (2021); Khaldi et
al. (2022), or only a small set of languages Nag et al. (2021).

To address this gap, and to incentivize research on supervised multilingual
RE, we introduce a multilingual version of one of the most prominent
supervised RE datasets, TACRED Zhang et al. (2017). MultiTACRED is created by
machine-translating TACRED instances and automatically projecting their entity
annotations. Machine translation is a popular approach for generating data in
cross-lingual learning Hu et al. (2020); Nag et al. (2021). Although the
quality of machine-translated data may be lower due to translation and
alignment errors Yarmohammadi et al. (2021), it has been shown to be
beneficial for classification and structured prediction tasks Hu et al.
(2020); Ozaki et al. (2021); Yarmohammadi et al. (2021).

The MultiTACRED dataset we present in this work covers 12 languages from 9
language families.111 MultiTACRED includes the following language families /
languages: German (Germanic); Finnish, Hungarian (Uralic); Spanish, French
(Romance); Arabic (Semitic); Hindi (Indo-Iranic); Japanese (Japonic); Polish,
Russian (Slavic); Turkish (Turkic); Chinese (Sino-Tibetan). We select
typologically diverse languages which span a large set of linguistic phenomena
such as compounding, inflection and pronoun-drop, and for which a monolingual
pretrained language model is available. We automatically and manually analyze
translation and annotation projection quality in all target languages, both in
general terms and with respect to the RE task, and identify typical error
categories for alignment and translation that may affect model performance. We
find that overall translation quality is judged to be quite good with respect
to the RE task, but that e.g. pronoun-dropping, coordination and compounding
may cause alignment and semantic errors that result in erroneous instances. In
addition, we experimentally evaluate fine-tuned pretrained mono- and
multilingual language models (PLM) in common training scenarios, using source
language (English), target language, or a mixture of both as training data. We
also evaluate an English data fine-tuned model on back-translated test
instances to estimate the effect of noise introduced by the MT system on model
performance. Our results show that in-language training works well, given a
suitable PLM. Cross-lingual zero-shot transfer is acceptable for languages
well-represented in the multilingual PLM, and combining English and target
language data for training considerably improves performance across the board.

To summarize, our work aims to answer the following research questions: Can we
reaffirm the usefulness of MT and cross-lingual annotation projection, in our
study for creating large-scale, high quality multilingual datasets for RE? How
do pretrained mono- and multilingual encoders compare to each other, in
within-language as well as cross-lingual evaluation scenarios? Answers to
these questions can provide insights for understanding language-specific
challenges in RE, and further research in cross-lingual representation and
transfer learning. The contributions of this paper are:

  * •

We introduce MultiTACRED, a translation of the widely used, large-scale TACRED
dataset into 12 typologically diverse target languages: Arabic, German,
Spanish, French, Finnish, Hindi, Hungarian, Japanese, Polish, Russian,
Turkish, and Chinese.

  * •

We present an evaluation of monolingual, cross-lingual, and multilingual
models to evaluate target language performance for all 12 languages.

  * •

We present insights into the quality of machine translation for RE, analyzing
alignment as well as language-specific errors.

##  2 Translating TACRED

Figure 1: Example translations from English to German, Polish, Turkish and
Chinese with XML markup for the head and tail entities to project relation
argument annotations.

We first briefly introduce the original TACRED dataset, and then describe the
language selection and automatic translation process. We wrap up with a
description of the analyses we conduct to verify the translation quality.

###  2.1 The TACRED dataset

The TAC Relation Extraction
Dataset222https://catalog.ldc.upenn.edu/LDC2018T24, under a LDC license,
introduced by Zhang et al. (2017), is a fully supervised dataset of sentence-
level binary relation mentions. It consists of 106k sentences with entity
mention pairs collected from the TAC
KBP333https://tac.nist.gov/2017/KBP/index.html evaluations 2009–2014, with the
years 2009 to 2012 used for training, 2013 for development, and 2014 for
testing. Each sentence is annotated with a head and a tail entity mention, and
labeled with one of 41 person- and organization-oriented relation types, e.g.
_per:title_ , _org:founded_ , or the label _no_relation_ for negative
instances. About 79.5% of the examples are labeled as _no_relation_.444The
first row of Table 6 in Appendix A summarizes key statistics of the dataset.
All relation labels were obtained by crowdsourcing, using Amazon Mechanical
Turk. Recent work by Alt et al. (2020) and Stoica et al. (2021) improved upon
the label quality of the crowd annotations by re-annotating large parts of the
dataset.

###  2.2 Automatic Translation

We translate the complete train, dev and test splits of TACRED into the target
languages, and in addition back-translate the test split into English to
generate machine-translated English test data. Each instance in the original
TACRED dataset is a list of tokens, with the head and tail entity arguments of
the potential relation specified via token offsets. For translation, we
concatenate tokens with whitespace and convert head and tail entity offsets
into XML-style markers to denote the arguments’ boundaries, as shown in Figure
1. We use the commercial services of
DeepL555https://api.deepl.com/v2/translate and
Google666https://translation.googleapis.com/language/translate/v3, since both
offer the functionality to preserve XML tag markup. Since API costs are
similar, we use DeepL for most languages, and only switch to Google for
languages not supported by DeepL (at the time we were running the MT). We
validate the translated text by checking the syntactic correctness of the XML
tag markup, and discard translations with invalid tag structure, e.g. missing
or invalid head or tail tag pairs.

After translation, we tokenize the translated text using language-specific
tokenizers.777See Appendix A for details. Finally, we store the translated
instances in same JSON format as the original TACRED English dataset, with
fields for tokens, entity types and offsets, label and instance id. We can
then easily apply the label corrections provided by e.g. Alt et al. (2020) or
Stoica et al. (2021) to any target language dataset by applying the respective
patch files.

We select target languages to cover a wide set of interesting linguistic
phenomena, such as compounding (e.g., German), inflection/derivation (e.g.,
German, Turkish, Russian), pronoun-dropping (e.g., Spanish, Finnish, Polish),
and varying degrees of synthesis (e.g., Turkish, Hungarian vs. Chinese). We
also try to ensure that there is a monolingual pretrained language model
available for each language, which is the case for all languages except
Hungarian. The final set of languages in MultiTACRED is: German, Finnish,
Hungarian, French, Spanish, Arabic, Hindi, Japanese, Chinese, Polish, Russian,
and Turkish. Table 6 in Appendix A lists key statistics per language.

###  2.3 Translation Quality Analysis

To verify the overall quality of the machine-translated data, we also manually
inspect translations. For each language, we randomly sample 100 instances from
the train split. For each sample instance, we display the source (English)
text with entity markup (see Figure 1 for the format), the target language
text with entity markup, and the relation label.

We then ask native speakers to judge the translations by answering two
questions: (Q1) Does the translated text meaningfully preserve the semantic
relation of the English original, regardless of minor translation errors?888If
necessary, human judges are first introduced to the task of relation
extraction. They are also given the list of relations and their official
definitions for reference. (Q2) Is the overall translation linguistically
acceptable for a native speaker? Human judges are instructed to read both the
English source and the translation carefully, and then to answer the two
questions with either yes or no. They may also add free-text comments, e.g. to
explain their judgements or to describe translation errors. The samples of
each language are judged by a single native speaker. Appendix B gives
additional details.

In addition, we conduct a manual analysis of the automatically discarded
translations, using a similar-sized random sample from the German, Russian and
Turkish _train_ splits, to identify possible reasons and error categories.
These analyses are performed by a single trained linguist per language, who is
also a native speaker of that language, with joint discussions to synthesize
observations. Results of both analyses are presented in Section 4.1.

##  3 Experiments

In this section, we describe the experiments we conduct to answer the research
questions “How does the performance of language-specific models compare to the
English original?” and “How does the performance of language-specific models
compare to multilingual models such as mBERT trained on the English source
data? How does the performance change when including target-language data for
training”. We first introduce the training scenarios, and then give details on
choice of models and hyperparameters, as well as the training process.

###  3.1 Training scenarios

We evaluate the usefulness of the translated datasets by following the most
prevalent approach of framing RE as a sentence-level supervised multi-class
classification task. Formally, given a relation set $\mathcal{R}$ and a text
$\bm{x}=[x_{1},x_{2},\ldots,x_{n}]$ (where $x_{1},\cdots,x_{n}$ are tokens)
with two disjoint spans $\bm{e}_{h}=[x_{i},\ldots,x_{j}]$ and
$\bm{e}_{t}=[x_{k},\ldots,x_{l}]$ denoting the head and tail entity mentions,
RE aims to predict the relation $r\in\mathcal{R}$ between $\bm{e}_{h}$ and
$\bm{e}_{t}$, or assign the no_relation class if no relation in $\mathcal{R}$
holds. Similar to prior work (e.g., Nag et al. (2021)), we evaluate relation
extraction models in several different transfer learning setups, which are
described next.

Monolingual We evaluate the performance of language-specific PLMs for each of
the 12 target languages, plus English, where the PLM is supervisedly fine-
tuned on the train split of the respective language.

Cross-lingual We evaluate the performance of a multilingual mBERT model on the
test split of each of the 12 target languages, plus English, after training on
the English train split.

Mixed / Multilingual We evaluate the performance of a multilingual mBERT model
on the test split of each of the 12 target languages, after training on the
complete English train split and a variable portion of the train split of the
target language, as suggested e.g. by Nag et al. (2021). We vary the amount of
target language data in {5%,10%,20%,30%,40%,50%,100%} of the available
training data. When using 100%, we are effectively doubling the size of the
training set, and “duplicating” each training instance.

Back-translation Finally, we also evaluate the performance of a BERT model
fine-tuned on the original (untranslated) English train split on the test sets
obtained by back-translating from each target language.

###  3.2 Training Details and Hyperparameters

We implement our experiments using the Hugging Face (HF) Transformers library
Wolf et al. (2020), Hydra Yadan (2019) and PyTorch Paszke et al. (2019).999We
make our code publicly available at https://github.com/DFKI-NLP/MultiTACRED
for better reproducibility. Due to the availability of pretrained models for
many languages and to keep things simple, we use BERT as the base PLM Devlin
et al. (2019).

We follow Baldini Soares et al. (2019) and enclose the subject and object
entity mentions with special token pairs, modifying the input to become
“[HEAD_START] subject [HEAD_END] …[TAIL_START] object [TAIL_END]”. In
addition, we append the entity types of subject and object to the input text
as special tokens, after a separator token: “…[SEP] [HEAD=type] [SEP]
[TAIL=type]”, where type is the entity type of the respective argument. We use
the final hidden state representation of the [CLS] token as the fixed length
representation of the input sequence that is fed into the classification
layer.

We train with batch size of 8 for 5 epochs, and optimize for cross-entropy.
The maximum sequence length is 128 for all models. We use AdamW with a
scenario-specific learning rate, no warmup, $\beta_{1}$ = 0.9,
$\beta_{2}=0.999$, $\epsilon=1e-8$, and linear decay of the learning rate.
Other hyperparameter values, as well as scenario-specific learning rates and
HF model identifiers for the pretrained BERT models, are listed in Appendix C.

We use micro-F1 as the evaluation metric, and report the median result of 5
runs with different, fixed random seeds. For all experiments, we use the
revised version of TACRED presented by Alt et al. (2020), which fixes a large
portion of the dev and test labels.101010Since both Alt et al. (2020) and
Stoica et al. (2021) provide fixes as patch files to the original dataset, it
is trivial to repeat our experiments using the original or the Re-TACRED
version of the data. We report scores on the test set in the respective target
language, denoted as $test_{L}$. Due to the automatic translation and
validation, training and test sets differ slightly across languages, and
absolute scores are thus not directly comparable across languages. We
therefore also report scores on the intersection test set of instances
available in all languages ($test_{\cap}$). This test set contains 11,874
instances, i.e. 76.6% of the original test set (see also Table 6).

##  4 Results and Discussion

We first present some insights into translation quality, and then discuss the
performance of models for the different training scenarios.

###  4.1 Translation Quality

Automatic validation As described in Section 2.2, we validate the target
language translation by checking whether the entity mention tag markup was
correctly transferred. On average, 2.3% of the instances were considered
invalid after translation. By far the largest numbers of such errors occurred
when translating to Japanese (9.6% of translated instances), followed by
Chinese (4.5%) and Spanish (3.8%). Table 6 in Appendix A gives more details,
and shows the number of valid translations for each language, per split and
also for the back-translation of the test split. Back-translation incurred
only half as many additional errors as compared to the initial translation of
the test split into the target language, presumably due to the fact that
‘hard’ examples had already been filtered out during the first translation
step.

The validation basically detects two types of alignment errors - missing and
additional alignments. An alignment may be missing in the case of pro-drop
languages, where the argument is not realized in the translation (e.g.
Spanish, Chinese), or in compound noun constructions in translations (e.g. in
German). In other cases, the aligner produces multiple, disjoint spans for one
of the arguments, e.g. in the case of coordinated conjunctions or compound
constructions with different word order in the target language (e.g. in
Spanish, French, Russian). Table 8 in Appendix D lists more examples for the
most frequent error categories we observed.

Language | Q1 (yes) | Q2 (yes)
---|---|---
ar | 85% | 92%
de | 100% | 91%
es | 78% | 91%
fi | 82% | 81%
fr | 92% | 93%
hi | 89% | 67%
hu | 89% | 48%
ja | 74% | 89%
pl | 73% | 93%
ru | 98% | 89%
tr | 99% | 90%
zh | 91% | 80%
Avg | 87.5% | 83.7%
Table 1: Translation quality, as judged by native speakers. (Q1) Does the
translated text meaningfully express the semantic relation of the English
original, regardless of minor translation errors? (Q2) Is the overall
translation linguistically acceptable for a native speaker?

Manual Validation Table 1 shows the results of the manual analysis of
translations. With regards to Q1, on average 87.5% of the translations are
considered to meaningfully express the relation, i.e. as in the original text.
Overall translation quality is judged to be good for 83.7% of the sampled
instances on average across languages. The most frequent error types noted by
the annotators are again alignment errors, such as aligning a random
(neighboring) token from the sentence with an English pronoun argument in
pronoun-dropping languages (e.g. Polish, Chinese), and non-matching spans
(inclusion/exclusion of tokens in the aligned span). Similar errors have also
been observed in a recent study by Chen et al. (2022b). In highly inflecting
languages such as Finnish or Turkish, the aligned entity often changes
morphologically (e.g. possessive/case suffixes).111111Inflection and
compounding both ideally could be solved by introducing alignment/argument
span boundaries at the morpheme level, but this in turn may raise issues with
e.g. PLM tokenization and entity masking. Other typical errors are
uncommon/wrong word choices, (e.g. due to missing or wrongly interpreted
sentence context), and the omission of parts of the original sentence. Less
frequent errors include atypical input which was not translated correctly
(e.g. sentences consisting of a list of sports results), and non-English
source text (approx. 1% of the data, see also Stoica et al. (2021)). Table 8
also lists examples for these error categories.

###  4.2 Model Performance

Test set | en | ar | de | es | fi | fr | hi | hu | ja | pl | ru | tr | zh
---|---|---|---|---|---|---|---|---|---|---|---|---|---
$test_{L}$ | 77.1 | 74.2 | 74.1 | 75.7 | 76.4 | 75.0 | 65.1 | 71.8 | 71.8 | 73.7 | 73.7 | 74.2 | 75.4
$test_{\cap}$ | 77.5 | 74.5 | 74.6 | 76.1 | 76.6 | 75.4 | 65.9 | 72.4 | 72.5 | 74.3 | 74.8 | 74.5 | 75.3
Table 2: Micro-F1 scores on the TACREV dataset for the monolingual setting.
The table shows the median micro-F1 score across 5 runs, on the test split of
the target language ($test_{L}$), and on the intersection of test instances
available in all languages ($test_{\cap}$).

Monolingual Table 2 shows the results for the monolingual setting. The English
BERT model achieves a reference median micro-F1 score of 77.1, which is in
line with similar results for fine-tuned PLMs. Alt et al. (2020); Chen et al.
(2022a); Zhou and Chen (2022) Micro-F1 scores for the other languages range
from 71.8 (Hungarian) to 76.4 (Finnish), with the notable exception of Hindi,
where the fine-tuned BERT model only achieves a micro-F1 score of
65.1121212See also Appendix C for an additional discussion of Hindi
performance issues. As discussed in Section 3.2, results are not directly
comparable across languages. However, the results in Table 2 show that
language-specific models perform reasonably well for many of the evaluated
languages.131313However, as various researchers have pointed out, model
performance may be over-estimated, since the models may be affected by
“translationese” Riley et al. (2020); Graham et al. (2020). Their lower
performance may be due to several reasons: translation errors, smaller train
and test splits because of the automatic validation step, the quality of the
pre-trained BERT model, as well as language-specific model errors.

Results on the intersection test set $test_{\cap}$ are slightly higher on
average, as compared to $test_{L}$. Relative differences to English, and the
overall ‘ranking’ of language-specific results, remain approximately the same.
This reaffirms the performance differences between languages observed on
$test_{L}$. It also suggests that the intersection test set contains fewer
challenging instances. For Hindi, these results, in combination with the low
manual evaluation score of 67% correct translations, suggest that the
translation quality is the main reason for the performance loss.

We conclude that for the monolingual scenario, machine translation is a viable
strategy to generate supervised data for relation extraction for most of the
evaluated languages. Fine-tuning a language-specific PLM on the translated
data yields reasonable results that are not much lower than those of the
English model for many tested languages.

Cross-lingual

Test set / Wikisize | Metric | en | ar | de | es | fi | fr | hi | hu | ja | pl | ru | tr | zh
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| P | 76.7 | 72.1 | 75.2 | 74.0 | 76.7 | 74.3 | 76.1 | 76.5 | 78.6 | 76.9 | 70.6 | 73.6 | 73.2
$test_{L}$ | R | 77.5 | 60.3 | 74.0 | 73.9 | 64.9 | 73.9 | 53.0 | 59.7 | 51.3 | 70.0 | 74.6 | 57.4 | 70.0
| F1 | 77.1 | 65.7 | 74.6 | 73.9 | 70.3 | 74.1 | 62.5 | 67.1 | 62.1 | 73.3 | 72.6 | 64.5 | 71.6
| P | 76.5 | 73.2 | 75.5 | 74.8 | 78.3 | 75.0 | 76.5 | 76.6 | 79.2 | 77.1 | 70.6 | 73.4 | 73.8
$test_{\cap}$ | R | 78.3 | 61.6 | 74.3 | 75.3 | 65.1 | 74.3 | 54.3 | 60.7 | 50.8 | 71.1 | 75.3 | 58.1 | 69.9
| F1 | 77.4 | 66.9 | 74.9 | 75.0 | 71.1 | 74.6 | 63.5 | 67.7 | 61.9 | 74.0 | 72.9 | 64.9 | 71.8
WikiSize |  $\log_{2}$(MB) | 14 | 10 | 12 | 12 | 9 | 12 | 7 | 10 | 11 | 11 | 12 | 9 | 11
Table 3: Micro-Precision, Recall and F1 scores on the TACREV dataset for the
cross-lingual setting. The table shows the median scores across 5 runs, on the
translated test split of the target language ($test_{L}$) and on the
intersection of test instances available in all languages ($test_{\cap}$),
when training mBERT on the English train split. For reference, the table also
shows the size of mBERT’s training data in a given language (Wikisize, as
$log_{2}$(MegaBytes), taken from Wu and Dredze (2020)). Languages with less
pretraining data in mBERT suffer a larger performance loss.

In the cross-lingual setting, micro-F1 scores are lower than in the
monolingual setting for many languages (see Table 3). The micro-F1 scores for
languages well-represented in mBERT’s pretraining data (e.g., English, German,
Chinese) are close to their monolingual counterparts, whereas for languages
like Arabic, Hungarian, Japanese, or Turkish, we observe a loss of 4.7 to 9.7
F1 points. This is mainly due to a much lower recall, for example, the median
recall for Japanese is only 51.3. The micro-F1 scores are highly correlated
with the pretraining data size of each language in mBERT: The Spearman rank
correlation coefficient of micro-F1 $L_{T}$ scores with the WikiSize reported
in Wu and Dredze (2020) is $r_{s}=0.82$ , the Pearson correlation coefficient
is $r_{p}=0.78$ . Hence, languages which are less well represented in mBERT’s
pretraining data exhibit worse relation extraction performance, as they don’t
benefit as much from the pretraining.

Precision, Recall and F1 on the intersection test set $test_{\cap}$ are again
slightly better on average than the scores on $test_{L}$. For Hindi, our
results reaffirm the observations made by Nag et al. (2021) for cross-lingual
training using only English training data. Our results for RE also confirm
prior work on the effectiveness of cross-lingual transfer learning for other
tasks (e.g., Conneau et al. (2020); Hu et al. (2020). While results are lower
than in the monolingual setting, they are still very reasonable for well-
resourced languages such as German or Spanish, with the benefit of incurring
no translation at all for training. However, for languages that are less well-
represented in mBERT, using a language-specific PLM in combination with in-
language training data produces far better results.

Mixed/Multilingual

In-lang data (%) | ar | de | es | fi | fr | hi | hu | ja | pl | ru | tr | zh | $\overline{\Delta}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---
- | 65.7 | 74.6 | 73.9 | 70.3 | 74.1 | 62.5 | 67.1 | 62.1 | 73.3 | 72.6 | 64.5 | 71.6 | -
5 | 68.7 | 74.9 | 73.9 | 70.5 | 74.3 | 67.0 | 68.8 | 69.2 | 72.2 | 73.0 | 67.8 | 72.5 | +1.7
10 | 69.0 | 74.5 | 73.7 | 70.4 | 73.6 | 68.0 | 68.9 | 70.6 | 72.0 | 72.7 | 68.9 | 73.0 | +1.9
20 | 71.0 | 74.4 | 74.5 | 72.2 | 73.9 | 69.9 | 70.2 | 71.7 | 73.3 | 73.3 | 69.2 | 73.0 | +2.9
30 | 71.4 | 74.8 | 74.8 | 72.3 | 74.2 | 70.1 | 71.0 | 72.3 | 72.9 | 73.2 | 70.1 | 73.7 | +3.2
40 | 71.2 | 74.3 | 74.5 | 72.1 | 73.9 | 70.4 | 70.8 | 71.6 | 73.0 | 73.1 | 70.3 | 74.0 | +3.1
50 | 71.2 | 74.7 | 74.4 | 73.0 | 74.4 | 71.8 | 70.9 | 72.6 | 73.1 | 73.3 | 70.3 | 74.8 | +3.5
100 | 73.5 | 75.8 | 75.9 | 73.5 | 75.6 | 72.4 | 72.4 | 73.3 | 74.3 | 75.6 | 71.6 | 75.4 | +4.7
Table 4: Micro-F1 scores on the TACREV dataset for the mixed/multilingual
setting. The table shows the median micro-F1 score across 5 runs, on the
translated test split of the target language, when training mBERT on the full
English train split and various portions, from 5% to 100%, of the translated
target language train split. The last column shows the mean improvement across
languages, compared to the cross-lingual baseline. Micro-F1 scores improve
when adding in-language training data for languages not well represented in
mBERT, while other languages mainly benefit when using all of the English and
in-language data, i.e. essentially doubling the amount of training data (last
row).

Table 4 shows the results obtained when training on both English and varying
amounts of target language data. We can observe a considerable increase of
mBERT’s performance for languages that are not well represented in mBERT’s
pretraining data, such as e.g. Hungarian. These languages benefit especially
from adding in-language training data, in some cases even surpassing the
performance of their respective monolingual model. For example, mBERT trained
on the union of the English and the complete Japanese train splits achieves a
micro-F1 score of 73.3, 11.2 points better than the cross-lingual score of
62.1 and 1.5 points better than the 71.8 obtained by the monolingual model on
the same test data. Languages like German, Spanish, and French don’t really
benefit from adding small amounts of in-language training data in our
evaluation, but show some improvements when adding 100% of the target language
training data (last row), i.e. when essentially doubling the size of the
training data. Other languages, like Finnish or Turkish, show improvements
over the cross-lingual baseline, but don’t reach the performance of their
monolingual counterpart.

Our results confirm observations made by Nag et al. (2021), who also find
improvements when training on a mixture of gold source language data and
projected silver target language data. For the related task of event
extraction, Yarmohammadi et al. (2021) also observe that the combination of
data projection via machine translation and multilingual PLMs can lead to
better performance than any one cross-lingual strategy on its own.

Back-translation Finally, Table 5 shows the performance of the English model
evaluated on the back-translated test splits of all target languages. Micro-F1
scores range from 69.6 to 76.1, and are somewhat lower than the score of 77.1
achieved by the same model on the original test set. For languages like
German, Spanish, and French, scores are very close to the original, while for
Arabic and Hungarian, we observe a loss of approximately 7 percentage points.
These differences may be due to the different quality of the MT systems per
language pair, but can also indicate that the model cannot always handle the
linguistic variance introduced by the back-translation.

Language | ar | de | es | fi | fr | hi | hu | ja | pl | ru | tr | zh
---|---|---|---|---|---|---|---|---|---|---|---|---
F1 | 69.6 | 76.1 | 75.8 | 73.6 | 75.9 | 73.3 | 70.0 | 72.2 | 74.7 | 74.0 | 72.1 | 74.8
Table 5: Median micro-F1 scores across 5 runs of the English BERT model
evaluated on the back-translated test splits of all languages. Compared to the
micro-F1 score of 77.1 on the untranslated English test set, back-translation
results are somewhat lower, due to MT system quality and the linguistic
variance introduced by the back-translation.

##  5 Related Work

Multilingual RE Datasets Prior work has primarily focused on the creation of
distantly supervised datasets. Dis-Rex Bhartiya et al. (2022) and RelX-Distant
Köksal and Özgür (2020) are large, Wikipedia-based datasets, but cover only 4
resp. 5 European languages. SMiLER Seganti et al. (2021) covers 14 European
languages, but is very imbalanced, both in terms of relation coverage in the
different languages and training data per language Chen et al. (2022c).

Manually supervised datasets include BizRel Khaldi et al. (2022), consisting
of 25.5K sentences labeled with 5 business-oriented relation types, in French,
English, Spanish and Chinese, and the IndoRE dataset of 32.6K sentences
covering 51 Wikidata relations, in Bengali, Hindi, Telugu and English Nag et
al. (2021). The IndoRE dataset uses MT to transfer manually labeled examples
from English to the three other languages, but implements a heuristic to
project entity annotations, without any verification step. Other datasets are
very small: The RelX dataset contains a manually translated parallel test set
of 502 sentences Köksal and Özgür (2020). Arviv et al. (2021) create a small
parallel RE dataset of 533 sentences by sampling from TACRED and translating
into Russian and Korean. For the related task of event extraction, datasets
worth mentioning are the multilingual ACE 2005 dataset Walker et al. (2006),
the TAC multilingual event extraction dataset Ellis et al. (2016), and the
work of Yarmohammadi et al. (2021).

Machine Translation for Cross-lingual Learning MT is a popular approach to
address the lack of data in cross-lingual learning Hu et al. (2020); Nag et
al. (2021). There are two basic options - translating target language data to
a well-resourced source language at inference time and applying a model
trained in the source language Asai et al. (2018); Cui et al. (2019); Hu et
al. (2020), or translating source language training data to the target
language, while also projecting any annotations required for training, and
then training a model in the target language Khalil et al. (2019);
Yarmohammadi et al. (2021); Kolluru et al. (2022). Both approaches depend on
the quality of the MT system, with translated data potentially suffering from
translation or alignment errors Aminian et al. (2017); Ozaki et al. (2021);
Yarmohammadi et al. (2021). With very few exceptions, using MT for
multilingual RE remains underexplored Faruqui and Kumar (2015); Zou et al.
(2018); Nag et al. (2021).

Multilingual RE Previous work in cross- and multilingual RE has explored a
variety of approaches. Kim et al. (2014) proposed cross-lingual annotation
projection, while Faruqui and Kumar (2015) machine-translate non-English
sentences to English, and then project the relation phrase back to the source
language for the task of Open RE. Verga et al. (2016) use multilingual word
embeddings to extract relations from Spanish text without using Spanish
training data. In a related approach, Ni and Florian (2019) describe an
approach for cross-lingual RE that is based on bilingual word embedding
mapping. Lin et al. (2017) employ convolutional networks to extract relation
embeddings from texts, and propose cross-lingual attention between relation
embeddings to model cross-lingual information consistency. Chen et al. (2022c)
introduce a prompt-based model, which requires only the translation of prompt
verbalizers. Their approach thus is especially useful in few- and zero-shot
scenarios.

##  6 Conclusion

We introduced a multilingual version of the large-scale TACRED relation
extraction dataset, obtained via machine translation and automatic annotation
projection. Baseline experiments with in-language as well as cross-lingual
transfer learning models showed that MT is a viable strategy to transfer
sentence-level RE instances and span-level entity annotations to typologically
diverse target languages, with target language RE performance comparable to
the English original for many languages.

However, we observe that a variety of errors may affect the translations and
annotation alignments, both due to the MT system and the linguistic features
of the target languages (e.g., compounding, high level of synthesis).
MultiTACRED can thus serve as a starting point for deeper analyses of
annotation projection and RE challenges in these languages. For example, we
would like to improve our understanding of RE annotation projection for highly
inflectional/synthetic languages, where token-level annotations are an
inadequate solution. In addition, constructing original-language test sets to
measure the effects of translationese remains an open challenge.

We plan to publish the translated dataset for the research community,
depending on LDC requirements for the original TACRED and the underlying TAC
corpus. We will also make publicly available the code for the automatic
translation, annotation projection, and our experiments.

## Limitations

A key limitation of this work is the dependence on a machine translation
system to get high-quality translations and annotation projections of the
dataset. Depending on the availability of language resources and the MT model
quality for a given language pair, the translations we use for training and
evaluation may be inaccurate, or be affected by translationese, possibly
leading to overly optimistic estimates of model performance. In addition,
since the annotation projection for relation arguments is completely
automatic, any alignment errors of the MT system will yield inaccurate
instances. Alignment is at the token-level, rendering it inadequate for e.g.
compounding or highly inflectional languages. Due to the significant resource
requirements of constructing adequately-sized test sets, another limitation is
the lack of evaluation on original-language test instances. While we manually
validate and analyze sample translations in each target language (Section 4.1)
for an initial exploration of MT effects, these efforts should be extended to
larger samples or the complete test sets. Finally, we limited this work to a
single dataset, which was constructed with a specific set of target relations
(person- and organization-related), from news and web text sources. These text
types and the corresponding relation expressions may be well reflected in the
training data of current MT systems, and thus easier to translate than
relation extraction datasets from other domains (e.g., biomedical), or other
text types (e.g., social media). The translated examples also reflect the
source language’s view of the world, not how the relations would necessarily
be formulated in the target language (e.g., use of metaphors, or ignorance of
cultural differences).

## Ethics Statement

We use the data of the original TACRED dataset “as is”. Our translations thus
reflect any biases of the original dataset and its construction process, as
well as biases of the MT models (e.g., rendering gender-neutral English nouns
to gendered nouns in a given target language). The authors of the original
TACRED dataset Zhang et al. (2017) have not stated measures that prevent
collecting sensitive text. Therefore, we do not rule out the possible risk of
sensitive content in the data. Furthermore, we utilize various BERT-based PLMs
in our experiments, which were pretrained on a wide variety of source data.
Our models may have inherited biases from these pretraining corpora.

Training jobs were run on a machine with a single NVIDIA RTX6000 GPU with 24
GB RAM. Running time per training/evaluation is approximately 1.5 hours for
the monolingual and cross-lingual models, and up to 2 hours for the
mixed/multilingual models that are trained on English and target language
data.

## Acknowledgements

We would like to thank David Harbecke, Aleksandra Gabryszak, Nils Feldhus and
the anonymous reviewers for their valuable comments and feedback on the paper.
We are also very grateful to all the helpful annotators who evaluated the
translations: Ammer Ayach, Yuxuan Chen, Nicolas Delinte, Aleksandra Gabryszak,
Maria Gonzalez Garcia, Elif Kara, Tomohiro Nishiyama, Akseli Reunamo, Kinga
Schumacher, Akash Sinha, and Tatjana Zeen. Finally, we’d like to thank Gabriel
Kressin and Phuc Tran Truong for their help with the code base and running the
translations and experiments. This work has been supported by the German
Federal Ministry for Economic Affairs and Climate Action as part of the
project PLASS (01MD19003E), and by the German Federal Ministry of Education
and Research as part of the projects CORA4NLP (01IW20010) and Text2Tech
(01IS22017B).

## References

  * Alt et al. (2020) Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020.  TACRED revisited: A thorough evaluation of the TACRED relation extraction task.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1558–1569, Online. Association for Computational Linguistics.
  * Aminian et al. (2017) Maryam Aminian, Mohammad Sadegh Rasooli, and Mona Diab. 2017.  Transferring semantic roles using translation and syntactic information.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 13–19, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Angeli et al. (2014) Gabor Angeli, Julie Tibshirani, Jean Wu, and Christopher D. Manning. 2014.  Combining distant and partial supervision for relation extraction.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1556–1567, Doha, Qatar. Association for Computational Linguistics.
  * Arviv et al. (2021) Ofir Arviv, Dmitry Nikolaev, Taelin Karidi, and Omri Abend. 2021.  On the relation between syntactic divergence and zero-shot performance.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4803–4817, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Asai et al. (2018) Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018.  Multilingual extractive reading comprehension by runtime machine translation.  _ArXiv_ , abs/1809.03275.
  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the Blanks: Distributional Similarity for Relation Learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy. Association for Computational Linguistics.
  * Bhartiya et al. (2022) Abhyuday Bhartiya, Kartikeya Badola, and Mausam . 2022.  DiS-ReX: A multilingual dataset for distantly supervised relation extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 849–863, Dublin, Ireland. Association for Computational Linguistics.
  * Chen et al. (2022a) Xiang Chen, Ningyu Zhang, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022a.  Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction.  _Proceedings of the ACM Web Conference 2022_.
  * Chen et al. (2022b) Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. 2022b.  Frustratingly easy label projection for cross-lingual transfer.  _CoRR_ , abs/2211.15613.
  * Chen et al. (2022c) Yuxuan Chen, David Harbecke, and Leonhard Hennig. 2022c.  Multilingual relation classification via efficient and effective prompting.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , Online and Abu Dhabi, the United Arab Emirates. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Cui et al. (2019) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019\.  Cross-lingual machine reading comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1586–1595, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Ellis et al. (2016) Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2016.  Overview of linguistic resources for the tac kbp 2016 evaluations: Methodologies and results.  In _Proceedings of TAC 2016_.
  * Faruqui and Kumar (2015) Manaal Faruqui and Shankar Kumar. 2015.  Multilingual open relation extraction using cross-lingual projection.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1351–1356, Denver, Colorado. Association for Computational Linguistics.
  * Graham et al. (2020) Yvette Graham, Barry Haddow, and Philipp Koehn. 2020.  Statistical power and translationese in machine translation evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 72–81, Online. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Khaldi et al. (2022) Hadjer Khaldi, Farah Benamara, Camille Pradel, Grégoire Sigel, and Nathalie Aussenac-Gilles. 2022.  How’s business going worldwide ? a multilingual annotated corpus for business relation extraction.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3696–3705, Marseille, France. European Language Resources Association.
  * Khalil et al. (2019) Talaat Khalil, Kornel Kiełczewski, Georgios Christos Chouliaras, Amina Keldibek, and Maarten Versteegh. 2019.  Cross-lingual intent classification in a low resource industrial setting.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6419–6424, Hong Kong, China. Association for Computational Linguistics.
  * Kim et al. (2014) Seokhwan Kim, Minwoo Jeong, Jonghoon Lee, and Gary Geunbae Lee. 2014.  Cross-lingual annotation projection for weakly-supervised relation extraction.  _ACM Transactions on Asian Language Information Processing_ , 13(1).
  * Köksal and Özgür (2020) Abdullatif Köksal and Arzucan Özgür. 2020.  The RELX dataset and matching the multilingual blanks for cross-lingual relation classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 340–350, Online. Association for Computational Linguistics.
  * Kolluru et al. (2022) Keshav Kolluru, Muqeeth Mohammed, Shubham Mittal, Soumen Chakrabarti, and Mausam . 2022.  Alignment-augmented consistent translation for multilingual open information extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2502–2517, Dublin, Ireland. Association for Computational Linguistics.
  * Lin et al. (2017) Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2017.  Neural relation extraction with multi-lingual attention.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 34–43, Vancouver, Canada. Association for Computational Linguistics.
  * Nag et al. (2021) Arijit Nag, Bidisha Samanta, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. 2021.  A data bootstrapping recipe for low-resource multilingual relation classification.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 575–587, Online. Association for Computational Linguistics.
  * Ni and Florian (2019) Jian Ni and Radu Florian. 2019.  Neural cross-lingual relation extraction based on bilingual word embedding mapping.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 399–409, Hong Kong, China. Association for Computational Linguistics.
  * Ozaki et al. (2021) Hiroaki Ozaki, Gaku Morio, Terufumi Morishita, and Toshinori Miyoshi. 2021.  Project-then-transfer: Effective two-stage cross-lingual transfer for semantic dependency parsing.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2586–2594, Online. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling Relations and Their Mentions without Labeled Text.  In _Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD ’10)_.
  * Riley et al. (2020) Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020.  Translationese as a language in “multilingual” NMT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7737–7746, Online. Association for Computational Linguistics.
  * Ruder et al. (2019) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019.  A survey of cross-lingual word embedding models.  _J. Artif. Int. Res._ , 65(1):569–630.
  * Seganti et al. (2021) Alessandro Seganti, Klaudia Firląg, Helena Skowronska, Michał Satława, and Piotr Andruszkiewicz. 2021.  Multilingual entity and relation extraction dataset and model.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1946–1955, Online. Association for Computational Linguistics.
  * Stoica et al. (2021) George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021\.  Re-tacred: Addressing shortcomings of the TACRED dataset.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 13843–13850. AAAI Press.
  * Verga et al. (2016) Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2016.  Multilingual relation extraction using compositional universal schema.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 886–896, San Diego, California. Association for Computational Linguistics.
  * Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006.  Ace 2005 multilingual training corpus.  Technical report, Linguistic Data Consortium.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_ , pages 120–130, Online. Association for Computational Linguistics.
  * Xie et al. (2021) Chenhao Xie, Jiaqing Liang, Jingping Liu, Chengsong Huang, Wenhao Huang, and Yanghua Xiao. 2021.  Revisiting the negative data of distantly supervised relation extraction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3572–3581, Online. Association for Computational Linguistics.
  * Yadan (2019) Omry Yadan. 2019.  Hydra - a framework for elegantly configuring complex applications.  Github.
  * Yarmohammadi et al. (2021) Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, and Benjamin Van Durme. 2021.  Everything is all it takes: A multipronged strategy for zero-shot cross-lingual information extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1950–1967, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 35–45, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zhou and Chen (2022) Wenxuan Zhou and Muhao Chen. 2022.  An improved baseline for sentence-level relation extraction.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 161–168, Online only. Association for Computational Linguistics.
  * Zou et al. (2018) Bowei Zou, Zengzhuang Xu, Yu Hong, and Guodong Zhou. 2018.  Adversarial feature adaptation for cross-lingual relation classification.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 437–448, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

##  Appendix A Translation Details

We use the following parameter settings for DeepL API calls:
split_sentences:1, tag_handling:xml, outline_detection:0. For Google, we use
format_:html, model:nmt.

Table 6 shows the number of syntactically valid and invalid translations for
each language and split, as well as for the back-translation of the test
split.

For tokenization, we use Spacy 3.2141414https://spacy.io with standard (non-
neural) models for de, es, fr, fi, ja, pl, ru, zh, and TranKIT
1.1.0151515https://github.com/nlp-uoregon/trankit for ar, hi, hu, tr.

The translation costs per language amount to approximately 460 Euro, for a
total character count of 22.9 million characters to be translated (source
sentences including entity markup tags), at a price of 20 Euro per 1 million
characters at the time of writing. Compared to an estimated annotation cost of
approximately 10K USD, translation costs amount to less than 5% of the cost of
fully annotating a similar-sized dataset in a new language.161616Stoica et al.
(2021) pay 0.15 USD per HIT of 5 sentences in TACRED. With an average of 3
crowd workers per HIT and a total of 106,264 examples in TACRED, this amounts
to approximately 9,564 USD. Angeli et al. (2014) report a cost of 3,156 USD
for annotating 23,725 examples, which would correspond to a cost of 14,135 USD
for the whole TACRED dataset.

| Train | Train Err | Dev | Dev Err | Test | Test Err | BT Test | BT Test Err
---|---|---|---|---|---|---|---|---
Language (Translation Engine) |  |  |  |  |  |  |  |
en (-) | 68,124 | - | 22,631 | - | 15,509 | - | - | -
ar (G) | 67,736 | 388 | 22,502 | 129 | 15,425 | 84 | 15,425 | 0
de (D) | 67,253 | 871 | 22,343 | 288 | 15,282 | 227 | 15,079 | 203
es (D) | 65,247 | 2,877 | 21,697 | 934 | 14,908 | 601 | 14,688 | 220
fi (D) | 66,751 | 1,373 | 22,268 | 363 | 15,083 | 426 | 14,462 | 621
fr (D) | 66,856 | 1,268 | 22,298 | 333 | 15,237 | 272 | 15,088 | 149
hi (G) | 67,751 | 373 | 22,511 | 120 | 15,440 | 69 | 15,440 | 0
hu (G) | 67,766 | 358 | 22,519 | 112 | 15,436 | 73 | 15,436 | 0
ja (D) | 61,571 | 6,553 | 20,290 | 2,341 | 13,701 | 1,808 | 12,913 | 805
pl (G) | 68,124 | 0 | 22,631 | 0 | 15,509 | 0 | 15,509 | 0
ru (D) | 66,413 | 1,711 | 21,998 | 633 | 14,995 | 514 | 14,703 | 292
tr (G) | 67,749 | 375 | 22,510 | 121 | 15,429 | 80 | 15,429 | 0
zh (D) | 65,260 | 2,864 | 21,538 | 1,093 | 14,694 | 815 | 14,021 | 681
$\cap_{all}$ | 54,251 | - | 17,809 | - | 11,874 | - | 9,944 | -
Table 6: MultiTACRED instances per language and split, and for the back-
translation (BT) of the test split. The _‘en’_ row shows the statistics of the
original TACRED. (G) and (D) refer to Google and DeepL, respectively. The
error columns list the number of instances discarded after translation due to
missing / erroneous entity tag markup. On average, 2.3% of the instances were
discarded due to invalid entity markup after translation. The last row shows
the intersection of valid instances available in all languages.

##  Appendix B Human Translation Analysis

For the manual analysis of translated TACRED instances, we recruited a single
native speaker for each language among the members of our lab and associated
partners. Annotators were not paid for the task, but performed it as part of
their work at the lab. All annotators are either Master’s degree or PhD
students, with a background in Linguistics, Computer Science, or a related
field. The full instructions given to annotators, after a personal
introduction to the task, are shown in Figure 2.

Figure 2: Task description given to human judges for translation quality
analysis.

##  Appendix C Additional Training Details

All pre-trained models evaluated in this study are used as they are available
from HuggingFace’s model hub, without any modifications. Our implementation
uses HF’s BertForSequenceClassification implementation with default settings
for dropout, positional embeddings, etc. Licenses for the pretrained BERT
models are listed in Table 7, if specified in the repository. The Transformers
library is available under the Apache 2.0 license, Hydra under the MIT
license, and PyTorch uses a modified BSD license.

For Hungarian, we use bert-base-multilingual-cased, since there is no
pretrained Hungarian BERT model available on the hub. For Hindi, we tried
several models by l3cube-pune, neuralspace-reverie, google and ai4bharat, but
all of these produced far worse results than the ones reported here for
l3cube-pune/hindi-bert-scratch. Interestingly, using bert-base-multilingual-
cased instead of l3cube-pune/hindi-bert-scratch as the base PLM produced far
better results for Hindi in the monolingual setting, at 71.1 micro-F1.

Language/Scenario | HuggingFace Model name | LR | License
---|---|---|---
ar | aubmindlab/bert-base-arabertv02 | 1e-5 | N/A
de | bert-base-german-cased | 3e-5 | MIT
en | bert-base-uncased | 3e-5 | Apache 2.0
es | dccuchile/bert-base-spanish-wwm-cased | 1e-5 | (CC BY 4.0)
fi | TurkuNLP/bert-base-finnish-cased-v1 | 7e-6 | N/A
fr | flaubert/flaubert_base_cased | 1e-5 | MIT
hi | l3cube-pune/hindi-bert-scratch | 7e-6 | CC BY 4.0
hu | bert-base-multilingual-cased | 1e-5 | Apache 2.0
ja | cl-tohoku/bert-base-japanese-whole-word-masking | 3e-5 | CC BY 4.0
pl | dkleczek/bert-base-polish-cased-v1 | 7e-6 | N/A
ru | sberbank-ai/ruBert-base | 3e-5 | Apache 2.0
tr | dbmdz/bert-base-turkish-cased | 1e-5 | MIT
zh | bert-base-chinese | 1e-5 | N/A
Cross-lingual mBERT | bert-base-multilingual-cased | 1e-5 | Apache 2.0
Multilingual mBERT | bert-base-multilingual-cased | 1e-5 | Apache 2.0
Table 7: Best learning rate and model identifiers per language for the
monolingual settings, and for the cross- and multilingual scenarios. The table
also lists the model license, if it was available.

We experimented with learning rates in $[3e-6,7e-6,1e-5,3-e5,5e-5]$. We used
micro-F1 on the dev set as the criterion for hyperparameter selection. Table 7
lists the best learning rates per language and scenario. We use a fixed set of
random seeds {1337, 2674, 4011, 5348, 6685} for training across the 5 runs.

##  Appendix D Translation Error Examples

Table 8 lists common error types we identified in the translations of TACRED
instances.

Error Type |  Source | Lang. |  Translation |  Comment
---|---|---|---|---
Alignment - Missing |  <H>He</H> also presided over the country ’s <T>Constitutional Council</T> […] | es |  También presidió el <T>Consejo Constitucional</T> del país […] |  Head not marked due to dropped pronoun
Alignment - Definite Article |  <T>JetBlue Airways Corp</T> spokesman <H>Bryan Baldwin</H> said […] | es |  <H>El</H> portavoz de<T>JetBlue Airways Corp</T> <H>, Bryan Baldwin</H>, dijo […] |  ‘El’ is marked as additional head span
Alignment - Split span |  New <T>York-based Human Rights Watch</T> ( HRW ) , […] snubbed an invitation to testify […] | es |  <T>Human Rights Watch</T> (HRW), con sede en Nueva <T>York</T>, […] rechazaron una invitación para testificar […] |  Translation of ‘York-based’ syntactically different, leading to split span
Alignment - Split Compound |  […] Russian <T>Foreign Ministry</T> spokesman Andrei Nesterenko said on Thursday , <H>RIA Novosti</H> reported. | fr |  […] a déclaré jeudi le porte-parole du <T>ministère</T> russe <T>des affaires étrangères</T>, Andrei Nesterenko, selon <H>RIA Novosti</H>. |  French word order for adjectives leads to split span of compound ‘Foreign Ministry’
Alignment - Compound |  […] Seethapathy Chander , Deputy Director General with <T>ADB</T> ’s <H>Private Sector Department</H>. | de |  […] Seethapathy Chander, stellvertretender Generaldirektor der <H>ADB-Abteilung für den Privatsektor</H>. |  German translation uses a compound noun combining head and ‘department’
Alignment - Missing |  <H>She</H> was vibrant , she loved life and <T>she</T> always had a kind word for everyone. | de |  <H>Sie</H> war lebhaft, sie liebte das Leben und hatte immer ein freundliches Wort für jeden. |  Multiple occurrences of same pronoun seem to confuse aligner
Alignment - Coordination |  <H>Christopher Bentley</H> , a spokesman for Citizenship and <T>Immigration Services</T> […] | es |  <H>Christopher Bentley</H>, un portavoz de <T>los Servicios de</T> Ciudadanía e <T>Inmigración</T> […] |  Coordinated conjuction in Spanish leads to split span
Alignment - Wrong |  She said when <H>she</H> got pregnant in <T>2008</T> […] | pl |  Powiedziała, że kiedy w <T>2008</T> r. <H>zaszła</H> w ciążę […] |  ‘got’ marked instead of dropped pronoun ‘she’
Alignment - Extended |  <T>Alaskans</T> last chose a Democrat for the presidency in 1964 , when they backed Lyndon B. Johnson by a 2-1 margin over <H>Barry Goldwater</H> . | zh |  <T>阿拉斯加人上</T>一次选择民主党人担任总统是在1964年，当时他们以2比1的优势支持林登-B-约翰逊，而不是<H>巴里-戈德华特</H>。 |  ‘last’ is included in tail span
Alignment - Partial |  In August , <H>Baldino</H> […] had taken a leave of absence from his posts as Cephalon ’s chairman and <T>chief executive</T> . | pl |  W sierpniu <H>Baldino</H> […] wziął urlop od pełnienia funkcji prezesa i <T>dyrektora general</T> nego firmy Cephalon. |  ‘nego’ should be part of the tail span and not be split off of the word ‘generalnego’
Alignment - Inflection |  Some of the people profiled are <T>ABC</T> president <H>Steve McPherson</H> , […] | fi |  Mukana ovat muun muassa <T>ABC:n</T> pääjohtaja <H>Steve McPherson</H>, […] |  Tail ’ABC:n’ includes genitive case marker in Finnish
Non-English Source |  Dari arah Jakarta/Indramayu , <T>sekitar</T> 2 km sebelum Pasar Celancang , tepatnya di sebelah Kantor Kecamatan Suranenggala terdapat Tempat Pelelangan Ikan ( <H>TPI</H> ) . | - |  - |  Source language is Indonesian, not English
Sentence split |  <H>Stewart</H> is not saying that a 1987-style stock market crash is on the immediate horizon , and <T>he</T> concedes that “ by many measures , stocks are n’t overpriced , even at recent highs . ” | tr |  <H>Stewart</H>, 1987 tarzı bir borsa çöküşünün hemen ufukta olduğunu söylemiyor ve <T>o</T> “ birçok önlemle , hisse senetlerinin aşırı fiyatlandırılmadığını bile kabul ediyor . son zirvelerde. ” |  ‘son zirvelerde’ erroneously separated by end-of-sentence period
Translation incomplete |  Outlined in a filing with the <H>Federal Election Commission</H> , <T>Obama</T> ’s suggestion is notable because … | de |  Der Vorschlag <T>Obamas</T> ist bemerkenswert, weil … |  Translation is missing first part and head span
Atypical input |  Browns 5-10 […] <T>Cowboys</T> 5-10 […] <H>Jaguars</H> 8-7 […] Total : 42-93 ( .311 ) Total : 58-74 ( .439 ) Total : 53-81 ( .396 ) | zh |  Browns 5-10 […] <T>Cowboys</T>5-10 […] <H>Jaguars</H>8-7 […] Total : 42-93 ( .311 ) 总数 : 58-74 ( .439 ) 总数 : 53-81 ( .396 ) |  Almost no translation due to atypical input
Table 8: Common error types of translated TACRED examples. The first half of
the table shows alignment errors that can be automatically detected, such as
missing or additional aligned spans in the translation. The second half shows
error types identified by human judges.