# Weyl-Invariant Gravity and the Nature of Dark Matter

Meir Shimon  School of Physics and Astronomy, Tel Aviv University, Tel Aviv
69978, Israel<EMAIL_ADDRESS>
###### Abstract

The apparent missing mass in galaxies and galaxy clusters, commonly viewed as
evidence for dark matter, could possibly originate from gradients in the
gravitational coupling parameter, $G$, and active gravitational mass,
$M_{act}$, rather than hypothetical beyond-the-standard-model particles. We
argue that in (the weak field limit of) a Weyl-invariant extension of General
Relativity, one can simply affect the change
$\Phi_{b}(x)\rightarrow\Phi_{b}(x)+\Phi_{DM}(x)$, where $\Phi_{b}$ is the
baryon-sourced potential and $\Phi_{DM}$ is the ‘excess’ potential. This is
compensated by gradients of $GM_{act}$ and a fractional increase of
$O(-4\Phi_{DM}(x))$ in the baryon density, well below current detection
thresholds on all relevant scales.

##  1 Introduction

Two key components of the standard model (SM) of cosmology, $\Lambda$CDM, are
dark energy (DE) and dark matter (DM), where evidence for DM on galactic and
cosmological scales abounds. It is commonly thought that observations favor a
type of ‘cold’ DM (CDM) on all relevant scales. CDM constitutes $\sim 80\%$ of
the non-relativistic (NR) matter in the universe, and together with DE makes
up the ‘dark sector’ of cosmology, accounting for $\sim 95\%$ of the energy
budget of the universe at present.

DM has reincarnated several times over the nearly century old history of the
modern era of cosmology [1-3]. The first indication for missing mass, i.e. DM,
in galaxy clusters was provided by the velocity dispersion of a handful of
galaxies in the Coma cluster [4, 5]. Five decades later it has been concluded
based on observations of rotation curves in galaxies that DM halos dominate
the mass budget on galactic scales [6]. At around the same time a $\sim$1 Kev
mass neutrino was proposed as the dominant component of the NR energy budget
on cosmological scales [7].

Obvious DM candidates are massive compact halo objects (MACHOs) such as
primordial black holes [8-11], but this possibility seems to be less popular
at present as compared to weakly interacting massive particles (WIMPs), e.g.
[12-19]. The WIMP paradigm is held by many as the most attractive
microphysical explanation for DM on galactic and extragalactic scales. Yet, no
definite detection had been made in any of the few dozen terrestrial DM
particle detection experiments.

Cold, warm, and hot DM candidates are distinguished by the impact they have on
structure formation hierarchy and are thus constrained by probes of the growth
of structure. The simplest hot DM candidates are already ruled out on galactic
scales. Warm DM candidates are thought to be either gravitinos or sterile
neutrinos [20-22] that were non-thermally produced unlike the leading CDM
particles which would account for WIMPs that are thermally produced. While CDM
is the leading candidate on Hubble down to galactic scales, certain issues
with CDM on dwarf galaxy scales seem to be more optimally addressed by fuzzy
DM, e.g. [23]. Mimetic DM is yet another DM model that provides a geometrical
(rather than material) interpretation for DM on cosmological scales, e.g.
[24]. An alternative geometrical interpretation is provided by Horava gravity
[25].

The allowed parameter space of DM particles ranges from ultralight to
superheavy, that can be either bosons or fermions, neutral or charged, stable
or unstable, interacting with DE or not, etc [26-29]. Inspired by
supersymmetry, the list of candidate DM particles is large, e.g. [30]. These
include: the neutralino, axion, sterile neutralino, garavitinos, axino,
Q-balls, etc [31-33]. The number of model parameters in the minimal
supersymmetric standard model (MSSM) exceeds 100, and under a few simplifying
assumptions this number can be considerably reduced to $\sim 20$. Despite this
ample freedom, there is yet no single candidate that fully explains the entire
spectrum of DM phenomena, especially on the smallest cosmological scales, e.g.
[34, 35]. In spite of extensive theoretical effort there is no viable unified
DM model.

With dozens of direct [36-39] and indirect [40-46] detection experiments, as
well as with the large hadron collider (LHC), a vigorous hunt for WIMPs, non-
WIMP or Kaluza-Klein states [47, 48] is underway. Present and future indirect
efforts aim at detecting either $\gamma$-ray emission from DM annihilation at
the Galactic center and dwarf galaxies or via high energy neutrinos from the
Sun [49-54].

Perhaps the first compelling alternative approach to explain galactic rotation
curves and velocity dispersion in galaxy clusters was modified Newtonian
dynamics (MOND) [55-58]. As a phenomenological model, MOND lacks a
relativistic description thus preventing its application on cosmological
scales. Relativistic theories of modified gravity (MOG) and mimetic gravity,
also aim at explaining galactic rotation curves with no recourse to particle
DM [59, 60]. These typically include both scalar, vector and tensor degrees of
freedom [61, 62].

An alternative Weyl-invariant (WI) relativistic theory – fourth-order Weyl
gravity – was later proposed to account for the anomalous rotation curves in
galaxies with no recourse to DM [63-66]. The proposed remedy is based on an
exact spherically symmetric static solution of the field equations. A similar
solution based on Weyl invariant scalar tensor (WIST) gravity was found in
[67]. However, it has not been demonstrated that these alternative approaches
provide compelling explanations in other systems, e.g. galaxy clusters, and in
more realistic (dynamical and non–symmetric) cases such as merging bullet-like
galaxy clusters. Other attempts to resolve the DM enigma include, in
particular, scalar-tensor (ST) theories that provide specific mechanisms with
falsifiable predictions.

The discovery of a few bullet-like merging galaxy clusters is seen by many as
a serious challenge to MOND and other alternatives to the DM paradigm [68] .
In these systems the center of gravity (probed by lensing) and the luminous
baryonic matter (observed through its X-ray emission) are clearly separated
and this is conventionally interpreted as evidence that the bulk mass of the
merging clusters is made of dissipasionless matter. However, the existence of
DM in these systems is deduced from lensing measurements (of background
quasars). The latter depends on transversal gradients of the gravitational
potential $\nabla_{\perp}\Phi$; the equivalent DM density profile
$\rho_{DM}(x)$, which is not directly measured, can be deduced from maps of
$\nabla_{\perp}\Phi$ (integrated along the line of sight).

Crucial to this procedure in particular, and to the CDM paradigm in general,
is the assumption that the gravitational coupling $G$ is a universal constant.
This has been established to a high level of precision essentially only with
regard to its temporal evolution, e.g. [69, 70]. However, more relevant to the
current work is a possible space-dependence of the Planck mass, $m_{P}\propto
G^{-1/2}$, and more specifically scale-dependence, if an underlying
translation invariance is assumed. In comparison, the observed scale-
dependence of DM phenomena is implicitly set in WIMP or fuzzy DM models by,
e.g. the particle mass or de Broglie wavelength, respectively. We note in
passing that even if the conservative interpretation of bullet-like clusters
is accepted at face value, the existence of these systems only implies that DM
cannot be fully accounted for by baryons, by no means does this prove that DM
is particulate.

##  2 WIST theory

Our universe is highly symmetric; the electroweak and strong interactions are
described by a $U(1)\otimes SU(2)\otimes SU(3)$ gauge symmetry, and the
underlying symmetry of general relativity (GR) is general coordinate
invariance. A hitherto hidden symmetry of GR could be WI. The ultimate test of
the validity of symmetries in physical theories is experiment, as well as
naturalness, falsifiability, etc.

While the assumptions that the Higgs mass scale $v\approx 246$ GeV and chiral
symmetry breaking mass scale $\Lambda_{QCD}\approx 220$ Mev are universally
fixed seem to work well for the SM of particle physics, a similar assumption
for $G$ may fail already on galactic scales (unless DM is invoked or gravity
is modified), although it is consistent with solar system observations. If $G$
is determined by a scalar field then the convention that the former is
universal obviously implies constancy of the latter. In the following, we
relax this assumption and allow variation of $GM_{act}$, where $M_{act}$ is
the active gravitational mass, in space and time.

The WIST theory considered here is obtained by replacing the spacetime metric
$g_{\mu\nu}$ with $\phi\phi^{*}g_{\mu\nu}$, along with the replacement
$1/(16\pi G)\rightarrow\phi\phi^{*}/6$, everywhere in the Einstein-Hilbert
(EH) action $\mathcal{I}_{EH}=\int[R/(16\pi
G)+\mathcal{L}_{M}]\sqrt{-g}d^{4}x$, where $\phi$ & $\phi^{*}$ are a scalar
field and its complex conjugate, $\mathcal{L}_{M}$ is the lagrangian density
of matter and $g$ is the metric determinant. The resulting ST theory is

$\displaystyle\mathcal{I}_{ST}=\int\left(\frac{1}{6}|\phi|^{2}R+\phi_{\mu}\phi^{*\mu}+\mathcal{L}_{M}(|\phi|,\\{\psi\\})\right)\sqrt{-g}d^{4}x,$
(1)

where $\phi_{\mu}\equiv\phi_{,\mu}$ and $\mathcal{L}_{M}$ is now allowed to
explicitly depend on $|\phi|$. All other fields are collectively denoted by
$\\{\psi\\}$.

The model has a global U(1) symmetry but we will not need the phase of the
scalar field here. The latter plays an important role in the very early
universe in a cosmological model based on Eq. (1) [71]. By construction, the
action is invariant under Weyl transformations, i.e.
$g_{\mu\nu}\rightarrow\Omega^{2}g_{\mu\nu}$,
$\mathcal{L}_{M}\rightarrow\Omega^{-4}\mathcal{L}_{M}$,
$\phi\rightarrow\phi/\Omega$, with $\Omega(x)$ a continuous but otherwise
arbitrary spacetime-dependent function. The theory described by Eq. (1) was
first considered in [72] for the case of real $\phi$ and $\mathcal{L}_{M}$
independent of $\phi$. This additional freedom to locally rescale fields is
essential to our construction; it implies that $\Omega(x)$ can be so chosen so
as to modify any single metric perturbation potential to obtain any desired
value at any point in spacetime, but at the cost of corresponding
modifications to $\phi$ (i.e. $G$), and $\mathcal{L}_{M}$ (that determine the
energy density of matter and its pressure). This is exactly the case we
consider below; neglecting any vector and tensor perturbation modes, and
assuming vanishing stress, metric perturbations of scalar type are described
by a single function $\Phi(x)$.

Variation of Eq. (1) with respect to the metric and $\phi^{*}$ results in the
following field equations, respectively

$\displaystyle\frac{|\phi|^{2}}{3}G_{\mu\nu}=T_{M,\mu\nu}+\Theta_{\mu\nu}$ (2)
$\displaystyle\frac{1}{6}\phi
R-\Box\phi+\frac{\partial\mathcal{L}_{M}}{\partial\phi^{*}}=0,$ (3)

where

$\displaystyle
3\Theta_{\mu\nu}\equiv\phi^{*}_{\mu;\nu}\phi-2\phi^{*}_{\mu}\phi_{\nu}-g_{\mu\nu}(\phi^{*}\Box\phi-\frac{1}{2}\phi_{\alpha}^{*}\phi^{\alpha})+c.c.$
(4)

Eq. (2) is a generalization of Einstein equations, and $\Theta_{\mu\nu}$ is an
effective contribution to the energy-momentum tensor essentially due to
gradients of $\phi$ (that determines $G$ and $M_{act}$). In the weak field
limit Eq. (2) results in a modified Poisson equation. Multiplying Eq. (3) by
$\phi^{*}$ and adding the result to its complex conjugate and to the trace of
Eq. (2) results in the constraint

$\displaystyle\phi^{*}\frac{\partial\mathcal{L}_{M}}{\partial\phi^{*}}+\phi\frac{\partial\mathcal{L}_{M}}{\partial\phi}=T_{M},$
(5)

i.e. only pure radiation $T_{rad}=0$ is consistent with WI gravity unless
$\mathcal{L}_{M}$ explicitly depends on $|\phi|$. Recalling that
$\mathcal{L}_{M}$ is a potential in $\phi$, and since
$\rho_{M}=\mathcal{L}_{M}$ (assuming a perfect fluid) it then follows that for
NR matter, $\rho_{NR}\propto\rho$, i.e. $M_{act}\propto\rho$, where $\rho$ is
the modulus of $\phi$. We mention in passing that no violation of the
equivalence principle ensues insofar inertial and passive gravitational masses
are fixed, which we indeed assume here. The notion of passive gravitational
masses is vacuous in the present theory and we only use the Newtonian parlance
for clarity. Irrespective of that, the equivalence principle has never been
directly tested beyond the solar system anyway, and obviously not directly
with DM particles.

The relevant combination for NR gravitating source is $GM_{act}$ which is
$\propto\rho^{-1}$. While $GM_{act}$ is spacetime-dependent, we assume that
the SM of particle physics is left unchanged (i.e. WI is not a symmetry
thereof), and inertial masses are universal constants.

##  3 DM in gravitationally-bound structures

A key to the following considerations is the fact that ‘DM’ is only probed via
the force its exerts, rather than $\rho_{DM}$. The latter, we recall, is
absent from $\mathcal{L}_{M}$ in Eq. (1). In contrast, density distribution of
baryonic matter, $\rho_{b}$, is directly observed through electromagnetic
emission, absorption and scattering processes. Throughout, we assume that DM
is not real matter and that its apparent attributes arise from gradients of
$GM_{act}$.

Although we focus on WIST, the following arguments are equally well valid in
any WI theory of gravitation. However, in general other theories do not have a
Poisson-like equation as their weak field limit, and thus $G$ is only an
effective notion, e.g. [73]. Although there is no scalar field in the
gravitational sector of forth-order Weyl gravity such a scalar (or an
effective scalar) field is expected to appear in $\mathcal{L}_{M}$. The latter
must scale $\propto\Omega^{-4}$ in any WI theory, and since there are no mass
scales in the theory, effective masses, e.g. the Planck mass or active
gravitational mass, scale $\propto\Omega^{-1}$.

Neglecting DM effects, assuming negligible stress and vanishing vector and
tensor modes, the spacetime appropriate for galaxies and galaxy clusters is
described (in the weak field limit) by the line element

$\displaystyle
ds^{2}=-(1-2\Phi_{b})dt^{2}-dtdx^{i}v_{i}+(1+2\Phi_{b})dx_{i}dx^{i},$ (6)

where Latin indices (denoting spatial coordinates) are summed over, $v_{i}\ll
1$ is a NR velocity and $\Phi_{b}\ll 1$ is the linear gravitational potential
that satisfies the relativistic generalization of Poisson equation with a
source term $\rho_{b}$. Here, $v_{i}$ is already a linear perturbation over
Minkowski spacetime. Other than the condition $\Phi_{b}\ll 1$ we make no
specific assumptions about the spacetime-dependence (symmetry) of $\Phi_{b}$
and the following results are therefore general.

Applying a Weyl transformation with $\Omega=1+\Phi_{DM}(x)$ and recalling that
$g_{\mu\nu}\rightarrow\Omega^{2}g_{\mu\nu}$ and $\phi\rightarrow\phi/\Omega$,
then at first order, the gravitational potential in Eq. (6) is shifted,
$\Phi_{b}\rightarrow\Phi_{b}+\Phi_{DM}$, at the “cost” of $GM_{act}\propto
1+\Phi_{DM}$ and a corresponding change in $\rho_{b}$ that we discuss below.

We assume that $\Phi_{DM}\ll 1$ on the relevant range of scales but is a
priori an arbitrary function of spacetime. It accounts for the excess
gravitational potential required to explain galactic scale observations.
Translation invariance further implies that $\Omega$ is a function of scale
rather than location. Specifically, in the spherically symmetric static case,
we could select the Navarro-Frenk-White (NFW) [74] gravitational potential
profile, $\Phi_{NFW}(r)\propto r^{-1}\ln(1+r/r_{s})$ (where $r_{s}$ is an
object-specific scale parameter), or any other profile, e.g. [75], as our
$\Phi_{DM}(r)$. The underlying assumptions about the microphysics that lead to
the specific r-dependence of $\Phi_{NFW}(r)$ (or any other $\Phi_{DM}(r)$
profile for that matter that fits observational data) are irrelevant to our
discussion. It only matters that the combination
$\Phi=\Phi_{b}(x)+\Phi_{DM}(x)$ provides a good fit to the data. The excess
gravitational potential $\Phi_{DM}$ is supported by gradients of $\phi$, i.e.
of $GM_{act}$, via the $\Theta_{\mu\nu}$ addition to the baryonic energy-
momentum tensor in Eq. (2). In this sense,
“$T_{DM,\mu\nu}$”$=\Theta_{\mu\nu}$. Its nonvanishing components in spherical
coordinates, linear in $\Phi_{DM}\ll 1$, are
$\Theta_{\eta}^{\eta}=\frac{\phi^{2}}{3}(\Phi_{rr}+\frac{2}{r}\Phi_{r})$,
$\Theta_{r}^{r}=\frac{2\phi^{2}}{3r}\Phi_{r}$ and
$\Theta_{\theta}^{\theta}=\Theta_{\varphi}^{\varphi}=\frac{\phi^{2}}{3}(\Phi_{rr}+\frac{1}{r}\Phi_{r})$,
where the ‘DM’ subscript was dropped and it is assumed that $\Phi=\Phi(r)$.
The pressure in the r-direction differs in general from the pressure in the
other two orthogonal directions. Therefore, the effective energy-momentum
tensor associated with DM does not correspond to a perfect fluid, and in any
case does not describe pressureless matter as does CDM. This is not a problem
since neither DM pressure nor DM density [and consequently not its equation of
state (EOS) $w_{DM}$] are directly observed. Again, in practice the CDM
paradigm with $w_{DM}\approx 0$ is only a means to obtain an adequate
$\Phi_{DM}(r)$ fit on these scales assuming the latter satisfies the Poisson,
Euler and continuity equations provided that G is a universal constant. These
equations are essentially obtained from Eq. (2) and its derivatives in the
more general case (varying $G$ and $M_{act}$) considered here. As emphasized
above, from the WI perspective any guess of $\Phi_{DM}(r)$ is equally
legitimate insofar as it provides a reasonably good fit to the data combined
with $\Phi_{b}$, much like $|\phi|^{2}=3/(8\pi G)=constant$ across the entire
universe and over Hubble time is an educated guess based on observations over
the relatively small solar system scales. However, a significant difference is
that in the latter convention DM is required (to make up for the convenient
$|\phi|^{2}=3/(8\pi G)=constant$ choice) which is widely believed to consist
of beyond-the-SM particles.

Aside from accounting for the apparent effects of DM, the only (potentially
observational) consequence of the $\Omega=1+\Phi_{DM}(x)$ choice in these
systems is that $\rho_{b}$ transforms as
$\rho_{b}\rightarrow\rho_{b}/(1+\Phi_{DM})^{4}\approx\rho_{b}(1-4\Phi_{DM})$.
Since typically $-\Phi_{DM}=O(10^{-4})$ in galaxies or $O(10^{-5})$ in galaxy
clusters, then $O(4\Phi_{DM})$ amounts to a tenth of a percent increase in
$\rho_{b}$ at most, too weak to be measured given the typically complicated
morphology and lumpiness of baryon distribution, temperature uncertainty,
current observational precision, etc, on these scales. Therefore, the
transformation $\Omega=1+\Phi_{DM}(x)$ can significantly modify the
gravitational potential, e.g. outside galaxy cluster and galaxy cores, while
making only a subtle change in the observable luminous matter density, thereby
making the case for a ‘non-particle DM’. This procedure could clearly be
applied to merging bullet-like clusters as well, where in this case
$\Phi_{DM}$ is expected to depend on both space and time, with no impact on
our general conclusion that DM may well be a non-WIMP, and non-MACHO-related
phenomenon on galactic and galaxy cluster scales.

The approach explored here to remove the need for DM on galactic and super-
galactic scales could be similarly employed on cosmological scales since
typically $\Phi=O(10^{-5})$ on these scales. However, a NR DM component seems
to still be required at the background level by a wealth of cosmological
probes. In the concordance cosmological model the NR component accounts for
$\sim 30\%$ of the total energy density at present, while baryons account for
only $\sim 5\%$; the remaining $\sim 25\%$ is believed to be in CDM.

Similar to the replacement of $g_{\mu\nu}$ with $\phi\phi^{*}g_{\mu\nu}$ that
leads to Eq. (1), replacing the metric with
$g^{\alpha\beta}\varphi_{\alpha}\varphi_{\beta}g_{\mu\nu}$ (along with the
constraint $g^{\alpha\beta}\varphi_{\alpha}\varphi_{\beta}=1$, where $\varphi$
is a real scalar field) in the EH action could be made, that leads to “mimetic
gravity”. The latter theory was proposed as a solution to the DM problem on
the largest cosmological scales assuming the homogeneous and isotropic
Friedmann-Robertson background metric [76, 77]. According to this picture the
additional energy provided by $\varphi_{\mu}$ mimics CDM with an EOS
$w_{mim}=0$. This does not contradict our proposed solution in gravitationally
bound structures for which the effective matter density and pressure do not
correspond with NR matter since the “missing mass” problems in these two
entirely different physical environments are of completely different nature;
$GM_{act}$ is indeed constant on cosmological scales as in the SM of
cosmology, but spatially varies on galactic scales.

##  4 Summary

The perspective advocated in the present work is that WI may well be a
symmetry of gravitation, and that while GR represents a convenient specific
choice of units (namely constant $G$ and $M_{act}$), it may have misled us to
think that some exotic form of DM is required to account for observations
(either in the form of MACHOs or WIMPs). A more prosaic solution to the DM
problem might be that $GM_{act}$ is actually scale-dependent.

In the context of the proposed ST theory gradients of the scalar field
(essentially of $GM_{act}$) may provide the required additional energy and
pressure for warping space to reproduce the observed gravitational pull caused
by hypothesized DM without recourse to DM particles. Specifically, we show
that DM manifestations could be fully accounted for through the innocuous
change $\Phi_{b}\rightarrow\Phi_{b}+\Phi_{DM}$ at the ‘cost’ of $O(\Phi_{DM})$
fractional variations in $GM_{act}$ and $\rho_{b}$ in the range of
$O(10^{-3})-O(10^{-4})$ on the relevant galactic and super-galactic scales,
levels that are too weak to be directly observed.

As the conformal factor $\Omega(x)=1+\Phi_{DM}(x)$ [and consequently
$\Phi_{DM}(x)$] is arbitrary, it could be in principle selected to better-fit
observations (on all relevant scales, including dwarf galaxy scales) than
would standard particle-induced CDM gravitational potential profiles do. We
note that at the cosmological background level GR endowed with WI is
equivalent to mimetic gravity, and thus contains DM-like energy density and
energy density perturbations with no recourse to DM particles.

It may seem that since the conformal factor $\Omega(x)=1+\Phi_{DM}(x)$,
essentially $GM_{act}(x)$, can always be so chosen to fit the data then the
theory is not falsifiable. However, by the same rationale the choice
$|\phi|^{2}=3/(8\pi G)=constant$ employed in the SM has already been falsified
by observations on galactic scales, and it is only saved by invoking an
unobservable $\rho_{DM}(x)$ component. It is somewhat ironic that if existing
DM candidates are not found, the CDM paradigm may still hold up with the only
consequence that the vast WIMP parameter space is narrowed down, while in
contrast, if WIMPs are found with properties that are sufficient for
adequately explaining observations, our proposed solution (as is any other
alternative solution) to the DM problem is essentially falsified.

We conclude that if gravitation is indeed locally scale-invariant, a symmetry
hidden by universally setting $|\phi|^{2}=3/(8\pi G)=constant$ in GR (largely
motivated by convenience, at least on galactic and cosmological scales), then
the currently favorite identification of DM with WIMPs may just be an artifact
of arbitrarily adopting a constant system of units.

## References

  1. [1]

Bertone G and Hooper D 2018 Rev. Mod. Phys. 90 045002

  2. [2]

Trimble V 1987 Annu. Rev. Astron. Astrophys. 25 425

  3. [3]

Peebles P. J. E 2017 Nat. Astron. 1 0057

  4. [4]

Zwicky F 1933 Helv. Phys. Acta 6 110

  5. [5]

Andernach H and Zwicky F 2017 arXiv:1711.01693

  6. [6]

Rubin V. C 1983 Science 220 1339

  7. [7]

Peebles P. J. E 1982 Astrophys. J. Lett. 263 L1

  8. [8]

Chapline G. F 1975 Nature 253 251

  9. [9]

Carr B, Kühnel F and Sandstad M 2016 Phys. Rev. D 94, 083504

  10. [10]

Carr B and Kühnel F 2020 Annu. Rev. Nucl. Part. Sci. 70

  11. [11]

Frampton P. H 2016 Mod. Phys. Lett. A 31 1650093

  12. [12]

Bergström L 2000 Rep. Prog. Phys. 63 793

  13. [13]

Griest K and Kamionkowski M 2000 Phys. Rep. 333 167

  14. [14]

Muñoz C 2004 Int. J. Mod. Phys. A 19 3093

  15. [15]

Bergström L 2009 New J. Phys. 11 105006

  16. [16]

Feng J. L 2010 Annu. Rev. Astron. Astrophys. 48 495

  17. [17]

Steigman G, Dasgupta B and Beacom J. F 2012, Phys. Rev. D 86 023506

  18. [18]

Roszkowski L, Sessolo E. M and Trojanowski S 2018 Rep. Prog. Phys. 81 066201

  19. [19]

Jungman G, Kamionkowski M and Griest K 1996, Phys. Rep. 267 195

  20. [20]

Adhikari R Agostini M Ky N. A. et al. 2017 J. Cosmol. Astropart. Phys. 2017
025

  21. [21]

Boyarsky A, Drewes M, Lasserre T, et al. 2019, Prog. Part. Nucl. Phys. 104 1

  22. [22]

Kusenko A 2009 Phys. Rep. 481 1

  23. [23]

Hu W, Barkana R and Gruzinov A 2000 Phys. Rev. Lett. 85 1158

  24. [24]

Chamseddine A. H and Mukhanov V 2013 J. High Energy Phys. JHEP11(2013)135

  25. [25]

Mukohyama S 2009 Phys. Rev. D 80 064005

  26. [26]

Chung D. J. H, Kolb E. W and Riotto A 1999 Phys. Rev. D 59 023501

  27. [27]

Ferreira E. G. M 2020 arXiv:2005.03254

  28. [28]

de Rújula A, Glashow S. L and Sarid U 1990 Nucl. Phys. B 333 173

  29. [29]

Dimopoulos S, Eichler D, Esmailzadeh R, et al. 1990, Phys. Rev. D 41, 2388

  30. [30]

Bertone G 2010 Particle Dark Matter : Observations, Models and Searches,
Edited by Gianfranco Bertone. Published by Cambridge University Press,
Cambridge, UK ; New York : 2010. ISBN: 9780521763684

  31. [31]

Marsh D. J. E 2016 Phys. Rep. 643 1

  32. [32]

Covi L, Kim J. E and Roszkowski L 1999 Phys. Rev. Lett. 82 4180

  33. [33]

Kusenko A and Shaposhnikov M 1998 Phys. Lett. B 418 46

  34. [34]

Bullock J. S and Boylan-Kolchin M 2017 Annu. Rev. Astron. Astrophys. 55 343

  35. [35]

Milgrom M 2020 Studies in the History and Philosophy of Modern Physics 71 170

  36. [36]

Aprile E, Aalbers J, Agostini F, et al. 2017 Phys. Rev. Lett. 119 181301

  37. [37]

Ahmed Z, Akerib D. S, Arrenberg S, et al. 2009 Phys. Rev. Lett. 102 011301

  38. [38]

Aprile E, Aalbers J, Agostini F, et al. 2016 J. Cosmol. Astropart. Phys. 2016
027

  39. [39]

Aalbers, J., Agostini, F., Alfonsi, M., et al. 2016 J. Cosmol. Astropart.
Phys. 2016 017

  40. [40]

Adriani O, Barbarino G. C, Bazilevskaya G. A, et al. 2009 Phys. Rev. Lett. 102
051101

  41. [41]

Geringer-Sameth A and Koushiappas S. M 2011, Phys. Rev. Lett. 107 241303

  42. [42]

Bergström L, Bringmann T and Edsjö J 2008 Phys. Rev. D 78 103520

  43. [43]

Bergström L, Ullio P and Buckley J. H 1998 Astropart. Phys. 9 137

  44. [44]

Ackermann M, Ajello M, Allafort A, et al. 2010 J. Cosmol. Astropart. Phys.
2010 025

  45. [45]

Cesarini A, Fucito F, Lionetto A, et al. 2004 Astropart. Phys. 21 267

  46. [46]

Abazajian K. N and Kaplinghat M 2012 Phys. Rev. D 86 083511

  47. [47]

Cheng H.-C, Feng J. L and Matchev K. T 2002 Phys. Rev. Lett. 89 211301

  48. [48]

Hooper D and Kribs G. D 2004 Phys. Rev. D 70

  49. [49]

Bertone G, Hooper D and Silk J 2005, Phys. Rep. 405 279

  50. [50]

Bergström L 2012 Ann. Phys. 524 479

  51. [51]

Arcadi G, Dutra M, Ghosh P, et al. 2018, Eur. Phys. J. C 78 203

  52. [52]

Ackermann M, Albert A, Anderson B, et al. 2015 Phys. Rev. Lett. 115 231301

  53. [53]

Ullio P, Bergström L, Edsjö J, et al. 2002, Phys. Rev. D 66 123502

  54. [54]

Abdo A. A, Ackermann M, Ajello M, et al. 2010 Astrophys. J. 712 147

  55. [55]

Milgrom M 1983 Astrophys. J. 270 365

  56. [56]

Milgrom M 1983 Astrophys. J. 270 371

  57. [57]

Milgrom M 2010 Invisible Universe 1241 139

  58. [58]

Bekenstein J and Milgrom M 1984 Astrophys. J. 286 7

  59. [59]

Moffat J. W and Rahvar S 2013 Mon. Not. R. Astron. Soc. 436 1439

  60. [60]

Myrzakulov R, Sebastiani L, Vagnozzi S, et al. 2016 Class. Quantum Gravity 33
125005

  61. [61]

Bekenstein J. D 2004 Phys. Rev. D 70 083509

  62. [62]

Moffat J. W 2006 J. Cosmol. Astropart. Phys. 2006 004

  63. [63]

Mannheim P. D and Kazanas D 1989 Astrophys. J. 342 635

  64. [64]

Mannheim P. D and O’Brien J. G 2011 Phys. Rev. Lett 106 121101

  65. [65]

Mannheim P. D and O’Brien J. G 2012 Phys. Rev. D 70 85 124020

  66. [66]

O’Brien J. G and Mannheim P. D 2012 Mon. Not. R. Astron. Soc. 421 1273

  67. [67]

Li Q and Modesto L 2020 Gravit. Cosmol. 26 99

  68. [68]

Clowe D, Bradač M, Gonzalez A. H, et al. 2006 Astrophys. J. Lett 648 L109

  69. [69]

Bertotti B, Iess L and Tortora P 2003 Nature 425 374

  70. [70]

Zhu W. W, Stairs I. H, Demorest, P. B, et al. 2015 Astrophys. J. 809 41

  71. [71]

Shimon M 2021, in preparation

  72. [72]

Deser S 1970 Ann. Phys. 59 248

  73. [73]

Mannheim P. D and Kazanas D 1994 Gen. Relativ. Gravit. 26 337

  74. [74]

Navarro J. F Frenk C. S and White S. D. M 1997 Astrophys. J. 490 493

  75. [75]

Merritt D, Graham A. W, Moore B, et al. 2006 Astron. J. 132 2685

  76. [76]

Chamseddine A. H and Mukhanov V 2013 J. High Energy Phys. 2013 135
JHEP11(2013)135

  77. [77]

Matsumoto J, Odintsov S. D and Sushkov S. V 2015 Phys. Rev. D 91 064062


# Blending LLMs into Cascaded Speech Translation: KIT’s Offline Speech
Translation System for IWSLT 2024

Sai Koneru, Thai-Binh Nguyen, Ngoc-Quan Pham, Danni Liu, Zhaolin Li,
Alexander Waibel, Jan Niehues

Karlsruhe Institute of Technology

<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) are currently under exploration for various
tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT),
and even End-to-End Speech Translation (ST). In this paper, we present KIT’s
offline submission in the constrained + LLM track by incorporating recently
proposed techniques that can be added to any cascaded speech translation.
Specifically, we integrate Mistral-7B111mistralai/Mistral-7B-Instruct-v0.1
into our system to enhance it in two ways. Firstly, we refine the ASR outputs
by utilizing the N-best lists generated by our system and fine-tuning the LLM
to predict the transcript accurately. Secondly, we refine the MT outputs at
the document level by fine-tuning the LLM, leveraging both ASR and MT
predictions to improve translation quality. We find that integrating the LLM
into the ASR and MT systems results in an absolute improvement of $0.3\%$ in
Word Error Rate and $0.65\%$ in COMET for tst2019 test set. In challenging
test sets with overlapping speakers and background noise, we find that
integrating LLM is not beneficial due to poor ASR performance. Here, we use
ASR with chunked long-form decoding to improve context usage that may be
unavailable when transcribing with Voice Activity Detection segmentation
alone.

Blending LLMs into Cascaded Speech Translation: KIT’s Offline Speech
Translation System for IWSLT 2024

Sai Koneru, Thai-Binh Nguyen, Ngoc-Quan Pham, Danni Liu, Zhaolin Li, Alexander
Waibel, Jan Niehues Karlsruhe Institute of Technology
<EMAIL_ADDRESS>

##  1 Introduction

This paper provides an overview of Karlsruhe Institute of Technology’s speech
translation (ST) system developed for the offline track of IWSLT 2024. We
participated in the constrained plus large language models (LLMs) condition,
focusing on the translation direction from English to German. Under this
condition, LLMs with parameters of around 7 billion are allowed, and they have
proven effective in many NLP tasks. One of the interesting aspects of this
condition is how one can effectively integrate them into ST systems.

In recent years, there has been a significant interest in developing several
open-sourced and medium-scale LLMs (Touvron et al., 2023; Jiang et al., 2023).
The adaptability of LLMs to diverse tasks, using techniques such as In-
Context-Learning Brown et al. (2020) or Parameter-efficient fine-tuning with
4-bit quantization Hu et al. (2021); Dettmers et al. (2024), enables their
exploitation even with limited resources.

With these recent advancements, exploiting LLMs for ST shows great promise and
offers several potential benefits. For instance, one common challenge in
Automatic Speech Recognition (ASR) is dealing with input noise, which can
often render it difficult to comprehend the speaker’s words. However, LLMs,
trained on vast amounts of data, may excel at predicting words compared to
decoders trained solely during ASR. Moreover, LLMs possess a richer vocabulary
and understanding of complex terminology that task-specific ASR systems may
lack. Motivated by these advantages, various studies have explored the
integration of LLMs into ASR (Chen et al., 2024; Pu et al., 2023), Machine
Translation (MT) (Koneru et al., 2023), and ST (Hu et al., 2024).

Figure 1: ASR Refinement: The ASR system generates a few candidate hypotheses
with beam search, and the LLM generates a new hypothesis based on all the
candidates as proposed in Chen et al. (2024). We use the top 5 candidates in
all our experiments.

Chen et al. (2024) employ the LLM to generate a new hypothesis based on the
N-best list of the ASR model. This strategy relies on the observation that
N-best lists tend to exhibit enough diversity, especially during uncertain
conditions, allowing accurate transcript prediction by examining the list. On
the other hand, for MT, Koneru et al. (2023) proposes leveraging the LLM to
automatically postedit translations by analyzing the source and hypothesis
documents to rectify contextual errors. Both approaches are system-agnostic
and have demonstrated successful enhancement of system quality. Furthermore,
it is also the case that cascaded systems are shown to be superior than end to
end systems in previous IWSLT findings and submissions making the leveraging
of LLMs easily compatible. (Agarwal et al., 2023; Liu et al., 2023).

Our system builds on these two approaches to effectively use the LLMs to
improve the cascaded ST pipeline by refining the intermediate outputs at both
ASR and MT while maintaining its modular structure. We utilize pre-trained
models to create the individual components and fine-tune them with the allowed
data. Specifically, we employ WavLM (Chen et al., 2022) and MBART50 (Liu et
al., 2020) to initialize the ASR, and NLLB-200 (3.3B) (Costa-jussà et al.,
2022) for the MT module. As for the LLM, we opt for Mistral 7B Instruction-
Tuned (Jiang et al., 2023), considering it to be the most recent model within
the allowable options.

We present our main findings below:

  * •

We demonstrate that LLMs can be tailored to enhance both ASR (Section 4.1) and
MT systems (Section 4.2), resulting in an absolute improvement of $0.3\%$ in
Word Error Rate and $0.65\%$ in COMET, respectively, on the tst2019 test set.

  * •

While we observe significant enhancements in in-domain scenarios, we find that
these techniques are not applicable in challenging scenarios (such as
Overlapping Speakers, Background noise, etc.) due to poor ASR performance.

  * •

We demonstrate that employing chunked long-form decoding222We derive the
terminology from this blog post. significantly improves ASR performance in
challenging scenarios, such as the case of the ITV dev set. Specifically, we
observe a decrease in the word error rate from $37.83\%$ to $30.98\%$

##  2 Data

This section describes the evaluation and training data we use in our
experiments. For evaluation, we report results on the tst2019 and ACLdev
(Salesky et al., 2023) test sets to compare with findings from previous works
(Anastasopoulos et al., 2021; Agarwal et al., 2023). We also use the EPTV
(European Parlament activities), Itv (TV Series), and Peloton (Fitness TV) dev
sets from the subtitling track consisting of overlapping speakers with
different accents to evaluate the ASR performance in challenging scenarios.

As the data conditions did not change from IWSLT23 to this year, we rely on
the data processed from last year’s submission (KIT’23) (Liu et al., 2023).
For the training data of ASR, we use the same system that used Common Voice
(Ardila et al., 2020), LibriSpeech (Panayotov et al., 2015), MuST-C v2 (Di
Gangi et al., 2019), TED-LIUM v3 (Hernandez et al., 2018), and VoxPopuli (Wang
et al., 2021).

While for MT fine-tuning, we use the cleaned training data from last year
created from the available parallel data. This includes Europarl v7 and v10
(Koehn, 2005), NewsCommentary v16, OpenSubtitles v2018 (Lison and Tiedemann,
2016), Tatoeba (Tiedemann, 2012), ELRC-CORDIS_News and TED2020 (Reimers and
Gurevych, 2020) and consists in total of 23 million sentence pairs. For the
rest of the paper, we refer to the full parallel data as seed and TED2020 as
in-domain.

##  3 Overview

In this section, we provide an overview of our proposed cascaded system,
detailing each individual component. First, the input audio is sent to the ASR
system, which undergoes segmentation, and N-best lists are generated for each
segmented utterance. Next, the top candidates in the N-best list are fed as
input to the LLM, which is trained to refine the ASR output and generate a
final ASR hypothesis. Following this, the final ASR hypotheses are passed on
to the sentence-level MT system, which produces translations. Finally, the
sentence-level automatic transcripts and translations are fed into another
adapted LLM, which automatically post-edits and generates a coherent document
translation of the talk.

###  3.1 Automatic Speech Recognition

We employed the ASR model from our previous year’s submission (Liu et al.,
2023), considering its effectiveness in transcribing the TED domain. For
initialization, we utilized WavLM and mBART50 for the encoder and decoder,
respectively, before fine-tuning on the ASR data described in Section 2.
However, we encountered below-par ASR performance on the challenging sets
EPTV, Itv, and Peloton.

We identified several issues that hindered the effectiveness of our ASR model
with these sets. Firstly, the model itself was trained on single-talker
datasets but inferred with multi-talker noisy datasets, leading to a mismatch
in data distribution. Secondly, our typical use of the SHAS model for audio
segmentation introduced challenges, as it sometimes missed segmentations and
overlooked segments containing human speech.

Data shift is difficult to handle when the training dataset has not changed
since last year. We focused more on handling the latter by incorporating long-
form decoding. The key idea is to better use context (at the text or signal
level) for decoding. The long audio file is chunked into smaller segments with
a small overlap between adjacent segments. The model is run over each chunk,
and the inferred text is joined at the strides by finding the longest common
sequence between overlaps.

###  3.2 ASR refinement

Figure 2: Document Level MT Refinement: The LLM trained to post-edit uses
sentence-level transcripts and translations to generate a final document-level
coherent and consistent translation.

Once we have generated the N-best list, we select the top 5 candidates and
utilize an LLM to produce the final hypothesis as shown in Figure 1. In this
step, we can adapt the LLM to the task using either few-shot prompting or LoRA
fine-tuning techniques. We choose to fine-tune the LLM with adapters based on
the findings from (Chen et al., 2024). However, it is crucial to train the LLM
under conditions that simulate the test environment, where it should fix
errors of our ASR output rather than on the whisper generated in Chen et al.
(2024).

To generate the dataset for fine-tuning, we perform inference on our in-domain
training data using the gold segmentation. We create pairs comprising the
N-best list and the corresponding reference. It is worth noting that we
utilized the same data to train the ASR system, which is not ideal. However,
resource constraints prevented us from following the augmentation procedure
that mitigates this, which we explain further in Section 3.4. Despite this
limitation, manual analysis revealed that the ASR did not memorize the
training data and produced similar N-best lists to those observed in the test
conditions.

Following this, we fine-tuned the Mistral 7B Instruction-tuned LLM (Jiang et
al., 2023) using QLoRA (Dettmers et al., 2024), to predict the gold reference
based on the top candidates (see the prompt format below). Importantly, we
chose not to shuffle the order of the top candidates when providing it in the
prompt, as doing so would eliminate the ranking information provided to the
LLM, which could be crucial for its performance.

    Punctuate and Post-edit the hypothesis
    based on the predictions:
    Hyp 1 <SS> Hyp 2 <SS> Hyp 3 ..
    Post-edited Hypothesis:
    Gold Reference

###  3.3 Machine Translation

For building the MT system, we leverage the strong pre-trained model NLLB 200
3.3B (Costa-jussà et al., 2022) that is allowed in the constrained plus LLM
track. We perform a two-step fine-tuning approach. Initially, we fine-tune the
model on the seed data to adapt it to the spoken language domain.
Subsequently, in the second step, we conduct in-domain fine-tuning on TED (in-
domain) data, given its significance as one of the primary test sets in the
offline track. Additionally, we implement checkpoint averaging to improve
generalization with the last 3 checkpoints.

####  3.3.1 Restoring Punctuations

It is important to note that the ASR outputs lack punctuation. Therefore, we
conducted experiments with two punctuators. First, we utilized the
punctuations generated from the LLM ASR refinement process described in
Section 3.2. Second, we employed a DeltaLM-based punctuation model, which was
utilized in our previous year’s submission (Liu et al., 2023). We observed
that while the punctuations generated by the LLM were semantically correct,
they often resulted in long sequences and led to a degradation in MT
performance. As a result, we decided to opt for the second choice and segment
the text into sentences using manually crafted rules.

| tst2019 | ACLdev2023
---|---|---
KIT’23 (Liu et al., 2023) ASR | 3.1 | 11.3
KIT’23 ASR + LLM Refine | 2.8 | 10.6
Table 1: ASR word error rate scores on tst2019 and ACLdev2023 test sets. $+$
LLM refine indicates that the N-best list was post-edited to generate the
final hypothesis.

###  3.4 Document-level Automatic Post-Editing

After translating the individual sentences with the fine-tuned NLLB, the
outputs are not coherent as they are translated in isolation. Moreover, any
ASR errors that might be fixed by observing the full document will be
translated incorrectly. To mitigate this, we perform an additional step of
document-level automatic post-editing using the source transcripts and
sentence translations shown in Figure 2.

Similar to the situation outlined in Section 3.2, we encountered a lack of
data for fine-tuning the LLM for document-level post-editing. Hence, we
adopted the approach proposed by Koneru et al. (2023) to create the dataset.
We divided the in-domain TED data into two halves, each containing English
audio, English transcript, and German translation. Subsequently, we fine-tuned
MT models on each half using the pre-trained models described in Sections 3.1
and 3.3. Following this, we conducted inference using the gold segmentation
with our ASR and MT models trained on one half to the other half. This
procedure generated a synthetic dataset with noisy ASR input, MT predictions,
and corresponding gold references, leveraging the provided segmentation in the
data.

We then use the synthetic dataset to create instances of document-level post-
editing. We go through each talk and divide the transcripts into chunks, each
chunk containing a maximum of 256 tokens corresponding to the LLM tokenizer.
Then for each chunk, we use the transcript, hypothesis and reference to
transform them into the format below and train the LLM to predict the gold
reference given the noisy transcript and sentence-level hypothesis.

    Noisy English Transcript:
    ASR Hyp 1 <SS> ASR Hyp 2 <SS> ....
    German Translations:
    MT Hyp 1 <SS> MT Hyp 2 <SS> ....
    Post-Edited German Translations:
    Ref 1 <SS> Ref 2 <SS> ....

We use the delimiter "<SS>" to align with the input and perform sentence-level
evaluation. Then, we again fine-tune the Mistral 7B Instruction-tuned LLM
(Jiang et al., 2023) using QLoRA (Dettmers et al., 2024), training it to
predict the gold reference given the noisy transcript and translations. We
employ the sliding window with payload strategy during decoding as described
in Koneru et al. (2023).

##  4 Results

###  4.1 Automatic Speech Recognition

To evaluate the benefit of the additional ASR refinement step described in
Section 3.2, we compare the word error rate of our ASR system before and after
post-editing, as shown in Table 1. The ASR performance improves in both cases,
with a higher absolute improvement observed in the ACLdev set. The LLM is
particularly beneficial in the ACLdev set, given that it contains terminology
from the scientific domain where the LLM excels. We also observe a relative
improvement of $10\%$ in the TED talks, indicating that ASR refinement is
beneficial.

Model | EPTV | ITV | Peloton
---|---|---|---
KIT’23 ASR | 26.43 | 37.83 | 18.93
KIT’23 ASR + Gold Seg | 16.84 | 37.21 | 25.88
KIT’23 ASR + long-form | 17.54 | 30.98 | 20.79
Seamless v2 (Barrault et al., 2023) | 40.94 | 56.94 | 43.47
Table 2: ASR word error rate scores on the EPTV, Peloton and ITV dev set. Best
scores for each set are highlighted in bold.

However, the performance of the same ASR system on the challenge set was below
par. We conducted additional ablation studies and present the results in Table
2 for the challenge dev sets. We compared last year’s ASR system with three
conditions: providing gold segmentation, utilizing long-form decoding, and
using the recently developed Seamless V2 (Barrault et al., 2023).

We observed that providing gold segmentation achieved a score of $16.84$,
demonstrating its crucial role in handling this challenging set for EPTV.
Moreover, long-form decoding significantly narrowed the gap, decreasing the
word error rate for both EPTV and ITV. Meanwhile, our ASR shows the best
performance for Peloton without any modifications. Additionally, we evaluated
Seamless to assess its robustness and found that its performance was severely
lacking in comparison.

Based on these results, we use the ASR with standard segmentation for TED and
Peloton test sets. For EPTV and ITV, we use the ASR system with long-form
decoding. We found that the LLM cannot refine the N-best list given the poor
WER of KIT’23 ASR for the latter test sets and generates long sequences with
repetitions for most utterances. Therefore, we do not perform any ASR or MT
LLM refinements for ITV and EPTV sets and generate translations with a
standard cascaded ST pipeline.

###  4.2 Cascaded Speech Translation

In this section, we evaluate the final quality of our cascaded ST using the
mwerSegmenter to realign the hypothesis with the reference segmentation. We
report results with BLEU (Papineni et al., 2002) and Chrf2 (Popović, 2015)
computed by Sacrebleu (Post, 2018). We also report the COMET (Rei et al.,
2022) score using the default model333Unbabel/wmt22-comet-da.

###  4.3 Two-step Fine-tuning

We presented a two-step fine-tuning approach to adapt our MT system in Section
3.3 to the target domain. We report the translation quality on tst2019 test
set with this approach (last row) and other models for comparison in Table 3.

Model | tst2019
---|---
| BLEU | Chrf2 | COMET
KIT’23 TED* | 28.4 | 58.8 | 78.87
NLLB 3.3B | 26.6 | 57.7 | 77.41
Seamless v2 | 25.5 | 57.0 | 76.65
NLLB 3.3B + Seed | 26.9 | 57.9 | 77.87
NLLB 3.3B + Seed + TED | 27.6 | 58.5 | 78.49

Table 3: MT scores using KIT’23 ASR as input calculated by resegmenting with
mwerSegmenter. * indicates an unconstrained system that was trained on the
same data sources but in more languages than what is allowed for IWSLT24. TED
indicates the model adapted for TED and not ACLdev which was the official
submission from KIT for IWSLT23

Model | tst2019 | ACLdev2023
---|---|---
BLEU | Chrf2 | COMET | BLEU | Chrf2 | COMET
NLLB 3.3 | 26.6 | 57.7 | 77.41 | 35.0 | 63.9 | 74.83
NLLB 3.3 Seed | 26.9 | 57.9 | 77.87 | 34.7 | 63.8 | 75.67
ASR Refine + NLLB 3.3 Seed | 27.3 | 58.3 | 78.32 | 36.1 | 65.0 | 77.59
ASR Refine + NLLB 3.3 Seed + TED | 28.3 | 58.8 | 78.98 | 34.8 | 63.7 | 77.25
ASR Refine + NLLB 3.3 Seed + TED + Doc APE | 28.7 | 59.1 | 79.63 | 36.4 | 64.5 | 78.64

Table 4: MT scores using KIT’23 ASR as input calculated by resegmenting with
mwerSegmenter. ASR Refine indicates an additional ASR refinement step with the
LLM. Seed and TED indicate fine-tuning the NLLB 3.3 with seed alone or a two-
step process with additional fine-tuning on TED. Doc APE indicates an LLM
post-editing refinement to generate a coherent and consistent document. Best
scores in each metric and test set are highlighted in bold.

Firstly, we observe that Seamless performs inferiorly to NLLB across all
translation metrics. Consequently, we proceeded with NLLB for further
experiments.

Subsequently, fine-tuning the seed parallel data improved quality across all
metrics, notably increasing the score from $77.41$ to $77.87$ in COMET.
Following this, with the assistance of second-step fine-tuning, we observed
further improvements, resulting in scores reaching $78.49$. However, it is
important to note that this system still lags behind last year’s submission,
which was specifically adapted to the TED domain. Nevertheless, it’s worth
highlighting that this system was trained across multiple languages, placing
it in the unconstrained condition for IWSLT24. Moreover, we could not
replicate a similar adaptation process for NLLB due to resource and time
constraints.

###  4.4 LLM Refinement

We proposed improving the ASR outputs and converting sentence-level to
document-level translations using fine-tuned LLMs. We evaluate the benefits of
the individual steps and report the results of our final cascaded ST system in
Table 4 on tst2019 and ACLdev2023 test sets.

First, the benefits of two-step fine-tuning, ASR refinement, and document
post-editing complement each other. Using KITs 23 ASR with NLLB 3.3 B as a
baseline, we obtained $77.41$ COMET in tst2019 test set. However, including
all enhancements led to a total improvement of $2.23$ COMET points.
Furthermore, the improvements are consistent with both lexical and neural
metrics.

Next, we observed that integrating LLMs provides significant benefits in the
ACLdev set compared to the TED dev sets. This is plausible due to scientific
terminology and accented speakers in the ACLdev set. Both of these challenges
are well-suited for LLMs, as the quality of the initial systems is sufficient
to utilize context and rectify mistakes reliably.

##  5 Conclusion

This system paper presented KIT’s submission for the offline track in the
constrained + LLMs condition, focusing on the English-to-German translation
direction. Using modular techniques, we successfully integrated LLMs into any
cascaded ST pipeline. Additionally, we highlighted the benefits of long-form
decoding in scenarios involving noisy and overlapping speech.

For future work, we aim to explore robust techniques for integrating LLMs that
can effectively handle challenging scenarios where ASR quality is sub-par.
Furthermore, the translation’s latency is quite high as it needs to call the
LLM twice. However, integrating quality estimation techniques to decide when
we need the LLM can limit the effects of the high latency problem.

## Acknowledgments

This work is partly supported by the Helmholtz Programme-oriented Funding,
with project number 46.24.01, named AI for Language Technologies, funding from
the pilot program Core-Informatics of the Helmholtz Association (HGF). It also
received partial support from the European Union’s Horizon research and
innovation programme under grant agreement No 101135798, project Meetween (My
Personal AI Mediator for Virtual MEETtings BetWEEN People). The work was
partly performed on the HoreKa supercomputer funded by the Ministry of
Science, Research and the Arts Baden-Württemberg and by the Federal Ministry
of Education and Research.

## References

  * Agarwal et al. (2023) Milind Agarwal, Sweta Agarwal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, et al. 2023.  Findings of the iwslt 2023 evaluation campaign.  Association for Computational Linguistics.
  * Anastasopoulos et al. (2021) Antonios Anastasopoulos, Ondřej Bojar, Jacob Bremerman, Roldano Cattoni, Maha Elbayad, Marcello Federico, Xutai Ma, Satoshi Nakamura, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Alexander Waibel, Changhan Wang, and Matthew Wiesner. 2021.  FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN.  In _Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)_ , pages 1–29, Bangkok, Thailand (online). Association for Computational Linguistics.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020.  Common voice: A massively-multilingual speech corpus.  In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_ , pages 4218–4222. European Language Resources Association.
  * Barrault et al. (2023) Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023.  Seamlessm4t-massively multilingual & multimodal machine translation.  _arXiv preprint arXiv:2308.11596_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2024) Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, and Eng-Siong Chng. 2024.  Hyporadise: An open baseline for generative speech recognition with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Dettmers et al. (2024) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024.  Qlora: Efficient finetuning of quantized llms.  _Advances in Neural Information Processing Systems_ , 36.
  * Di Gangi et al. (2019) Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019.  MuST-C: a Multilingual Speech Translation Corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hernandez et al. (2018) François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia A. Tomashenko, and Yannick Estève. 2018.  TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation.  In _Speech and Computer - 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings_ , volume 11096 of _Lecture Notes in Computer Science_ , pages 198–208. Springer.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Hu et al. (2024) Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, and Eng Siong Chng. 2024.  Gentranslate: Large language models are generative multilingual speech and machine translators.  _arXiv preprint arXiv:2402.06894_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: Papers_ , pages 79–86, Phuket, Thailand.
  * Koneru et al. (2023) Sai Koneru, Miriam Exel, Matthias Huck, and Jan Niehues. 2023.  Contextual refinement of translations: Large language models for sentence and document-level post-editing.  _arXiv preprint arXiv:2310.14855_.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Liu et al. (2023) Danni Liu, Thai Binh Nguyen, Sai Koneru, Enes Yavuz Ugan, Ngoc-Quan Pham, Tuan-Nam Nguyen, Tu Anh Dinh, Carlos Mullov, Alexander Waibel, and Jan Niehues. 2023.  Kit’s multilingual speech translation system for iwslt 2023.  _arXiv preprint arXiv:2306.05320_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Mangrulkar et al. (2022) Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022.  Peft: State-of-the-art parameter-efficient fine-tuning methods.  https://github.com/huggingface/peft.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North_. Association for Computational Linguistics.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An ASR corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015_ , pages 5206–5210. IEEE.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Pu et al. (2023) Jie Pu, Thai-Son Nguyen, and Sebastian Stüker. 2023.  Multi-stage large language model correction for speech recognition.  _arXiv preprint arXiv:2310.11532_.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Salesky et al. (2023) Elizabeth Salesky, Kareem Darwish, Mohamed Al-Badrashiny, Mona Diab, and Jan Niehues. 2023.  Evaluating Multilingual Speech Translation Under Realistic Conditions with Resegmentation and Terminology.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2021) Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021.  VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 993–1003, Online. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.

##  Appendix A Appendix

We use the transformers library (Wolf et al., 2019) for fine-tuning our ASR
and LLM and the fairseq toolkit (Ott et al., 2019) for fine-tuning NLLB 3.3B.
For the ASR training, we set the batch size to $384$, resulting in
approximately 128 minutes per batch. We employ a warmup strategy over $2,000$
steps and a total of $100,000$ training steps. The learning rate is
initialized to $1e-4$.

For the NLLB fine-tuning experiments, we use a learning rate set to $5e-5$,
label smoothing to $0.1$, drop out to $0.1$, attention drop-out to $0.1$. We
use the Adam optimizer with betas to $(0,9,0.98)$ and the remaining optimizer
parameters to default. We used a batch size of maximum $3096$ making one step,
update-freq to $16$ and validating on the dev set after every epoch. We
stopped the training after the dev loss did not increase after 10 epochs. For
fine-tuning the LLM with QLoRA we use the peft (Mangrulkar et al., 2022) along
with the transformers library. We add LoRA adapters to the target modules
[q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]. We set the
adapter rank to $16$, alpha to $32$ and lora dropout to $0.1$. We use a batch
size of $8$, learning rate of $5e-5$ with other parameters set to default.
After every 200 steps, we validate and terminate the training if it does not
improve 10 consecutive times.

During inference, we use beam search for all ASR, MT and LLM components. The
ASR and MT decode with beam size of 5, whereas the LLM does it with beam size
of $3$.