# Nonconservative unimodular gravity: Gravitational waves

Júlio C. Fabris<EMAIL_ADDRESS>Núcleo Cosmo-ufes & Departamento
de Física, Universidade Federal do Espírito Santo (UFES)
Av. Fernando Ferrari, 540, CEP 29.075-910, Vitória, ES, Brazil.  National
Research Nuclear University MEPhI, Kashirskoe sh. 31, Moscow 115409, Russia
Marcelo H. Alvarenga<EMAIL_ADDRESS>Núcleo Cosmo-ufes &
Departamento de Física, Universidade Federal do Espírito Santo (UFES)
Av. Fernando Ferrari, 540, CEP 29.075-910, Vitória, ES, Brazil.  Mahamadou
Hamani Daouda<EMAIL_ADDRESS>Département de Physique - Université de
Niamey, Niamey, Niger.  Hermano Velten<EMAIL_ADDRESS>Departamento de Física, Universidade Federal de Ouro Preto (UFOP), Campus
Universitário Morro do Cruzeiro, 35.400-000, Ouro Preto, Brazil

###### Abstract

Unimodular gravity is characterized by an extra condition with respect to
General Relativity: the determinant of the metric is constant. This extra
condition leads to a more restricted class of invariance by coordinate
transformation. Even though, if the conservation of the energy-momentum tensor
is imposed in unimodular gravity, the General Relativity theory is recovered
with an additional integration constant which is associated to the
cosmological term $\Lambda$. However, if the energy-momentum tensor does not
conserve separately, a new geometric structure appears with potentially
observational signatures. In this text, we consider the evolution of
gravitational waves in the nonconservative unimodular gravity, showing how it
differs from the usual signatures in the standard model.

##  I Introduction

General Relativity (GR) is considered as the modern theory of gravitation
since its final formulation in 1915. However, soon after the GR theory has
been introduced many other theories for the gravitational interaction have
been proposed. Unimodular gravity is one of the oldest alternative to GR, see
for example Ref. wein-a  for a discussion on this alternative formulation of a
gravity theory. Unimodular gravity is similar to GR in many respects. One
important similarity is that both are geometric theories of gravity based on a
Lagrangian containing the Ricci scalar. However, unimodular gravity has one
special new ingredient: the determinant of the metric is imposed to be a
constant which can be set equal to 1. In practise, this leads to equations
that are traceless. Then, source of the geometrical structure, i.e., the right
hand side of the field equations, must also correspond to a traceless
structure.

In GR theory, the Bianchi identities lead automatically to conservation of the
energy-momentum tensor. This is connected to the invariance of the theory by
general diffeomorphisms. The unimodular gravity has a more restricted class of
transformation due to the condition on the determinant of the metric, usually
called transverse diffeomorphism. For a detailed discussion on the transverse
diffeomorphism see for example Ref. trans  and references therein. The
invariance by this restricted class of transformations, with respect to the
general diffeomorphism, in the unimodular gravity has one particular
consequence for the divergence of the energy-momentum tensor: it can be zero,
implying the usual conservation laws, and the GR equations are recovered with
an extra integration constant which can be identified with the cosmological
constant; or the energy-momentum tensor does not conserve, and a radical new
structure appears. In this article, we want to explore the second option as
described later.

The unimodular condition on the determinant of the metric implies that volumes
must be preserved in a coordinate transformation. This property, together with
the other aspects related to the notion of transverse diffeomorphism, has been
explored in many studies related to quantum gravity quantum1 . See also Ref.
Herrero-Valea:2018ilg  whose results are more relevant to our work.

The unimodular equations are traceless but not conformal invariant. Traceless
equations in gravity theory have been evoked many times in the literature, see
for example Ref. ellis . In Ref. mimetic1 ; mimetic2  the mimetic gravity
theory has been proposed, which may be written also in a traceless form, and
which have many interesting consequences for the early and late cosmological
scenarios. Here we will restrict ourselves to the unimodular gravity. In
general, in the investigations involving unimodular gravity, the conservation
of the energy-momentum tensor is imposed. This imposition leads to some nice
properties. One of them is that the theory becomes equivalent to GR in
presence of a cosmological term that appears an integration constant. In some
sense the diffeomorphism invariance may be recovered, see for example the
discussion in Ref. wein-a .

However, the second option described above, the non conservation of the
energy-momentum tensor, may open many interesting possibilities. One of them
is that the resulting cosmological model is equivalent, at the background
level, to cosmological model in GR with only radiation and a cosmological
constant. This can be seen as a bad feature, since in the standard
cosmological model based in the GR theory a matter dominated phase is
necessary in order to allow for a successful large structure formation
process. But, in the unimodular gravity the perturbative features may allow
structure formation even in a radiative universe. This problem is treated more
in detail in a companion paper marcelo .

In the present paper we have a very specific goal: to study the evolution of
gravitational waves in a cosmological model resulting from unimodular gravity
with non conservation of the energy-momentum tensor. We will call it
nonconservative unimodular gravity. We will verify that gravitational waves in
this nonconservative context have specific features that may allow for testing
the model with future observational data on primordial gravitational waves.
Our analysis will be fundamentally qualitative for reasons to be discussed
later in this article, but opens possibilities for a more quantitative study.

The paper is organized as follows. In the next section we outline the main
features of the unimodular gravity. In section III we discusss the
construction of the nonconservative unimodular gravity. In section IV we
compute the evolution of the cosmological gravitational waves in the
nonconservative unimodular gravity and in section V we compare it with know
results in the standard cosmological model. In section VI we discuss our main
results and present our conclusions.

##  II Unimodular gravity

In order to better evaluate the meaning of the unimodular condition let us
briefly first review aspects of the structure of GR theory. Here we follow
closely a similar exposition made in Ref. brand .

The equations of GR are obtained through variational principle from the
Einstein-Hilbert action,

$\displaystyle{\cal S}=\int d^{4}x\sqrt{-g}\\{R-2\Lambda+{\cal L}_{m}\\},$ (1)

where ${\cal L}_{m}$ denotes the matter Lagrangian density, $\Lambda$ is the
cosmological constant and $R$ is the Ricci scalar. This is the most simple
action allowed in four dimensions leading to second order differential
equations. We have already introduced $\Lambda$ in the action since it is
allowed by the Lovelock theorem. The presence of the cosmological term may
also allow to implement later the standard cosmological model for the
universe, the $\Lambda$CDM model.

The variation of the action (1) with respect to the metric $g_{\mu\nu}$ leads
to the GR equations in presence of matter and a cosmological constant:

$\displaystyle R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}R=8\pi
GT_{\mu\nu}+g_{\mu\nu}\Lambda.$ (2)

Here, the energy-momentum tensor is defined as,

$\displaystyle T_{\mu\nu}=-\frac{2}{\sqrt{-g}}\frac{\delta\sqrt{-g}{\cal
L}_{m}}{\delta g^{\mu\nu}}.$ (3)

The use of the Bianchi identities lead the conservation of the energy-momentum
tensor,

$\displaystyle{T^{\mu\nu}}_{;\mu}=0.$ (4)

This property is directly connect with the invariance of the GR theory with
respect to difeomorphism, see Ref. wald .

Now we turn to unimodular gravity. In principle, we must follow the same
procedure used to obtain the GR equations, but implementing at the same time
the unimodular condition, $g=$ constant (constant that may be equal to 1). In
general, the introduction of a constraint in an action can be made with the
help of Lagrange’s multiplier. In doing so, in the unimodular case, some
details must be outlined. Let us consider, for example, the unimodular
condition for a flat, isotropic and homogenous universe, described by the
metric,

$\displaystyle ds^{2}=N^{2}dt^{2}-a(t)^{2}(dx^{2}+dy^{2}+dz^{2}),$ (5)

where $N$ is the lapse function and $a(t)$ is the scale factor. In this case,
the unimodular condition, with the constant equal to one, implies,

$\displaystyle Na^{3}=1.$ (6)

Hence, the unimodular condition would fix the lapse function, $N=a^{-3}$. We
will see below how to circumvent this limitation.

In order to construct the unimodular equations we use, as already stated, the
Lagrangian multipliers in the action: The action reads,

$\displaystyle{\cal S}=\int
d^{4}x\biggr{\\{}\sqrt{-g}R-\chi(\sqrt{-g}-1)\biggl{\\}}+\int
d^{4}x\sqrt{-g}{\cal L}_{m},$ (7)

where all the quantities are defined as in GR, and $\chi$ is the Lagrangian
multiplier. To circumvent the restriction on the lapse function described
above, we modify the action (7), rewriting it as,

$\displaystyle{\cal S}=\int
d^{4}x\biggr{\\{}\sqrt{-g}R-\chi(\sqrt{-g}-\xi)\biggl{\\}}+\int
d^{4}x\sqrt{-g}{\cal L}_{m}.$ (8)

where $\xi=\xi(t)$ is an arbitrary function of time which is not subjected to
the variational principle: $\xi(t)$ may be considered as an external field.

Varying the action (8) with respect to the metric, it follows,

$\displaystyle R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}R+\frac{\chi}{2}g_{\mu\nu}=8\pi
GT_{\mu\nu}.$ (9)

Varying with respect to $\chi$ we obtain the unimodular constraint.

$\displaystyle\xi=\sqrt{-g}.$ (10)

Since, $\xi$ is an arbitrary function of time, now any choice for the lapse
function is possible, as in GR.

From the trace of (9) we obtain

$\displaystyle\chi=\frac{R}{2}+8\pi G\frac{T}{2}.$ (11)

Inserting this result in (9) we obtain,

$\displaystyle R_{\mu\nu}-\frac{1}{4}g_{\mu\nu}R=8\pi
G\biggr{(}T_{\mu\nu}-\frac{1}{4}g_{\mu\nu}T\biggl{)}.$ (12)

These are the unimodular field equations. Remark that the equations are
traceless.

Now we inspect the conservation laws. The use of the Bianchi identities lead
to the following new conservation law:

$\displaystyle\frac{R^{;\nu}}{4}=8\pi
G\biggr{(}{T^{\mu\nu}}_{;\mu}-\frac{1}{4}T^{;\nu}\biggl{)}.$ (13)

In principle, the GR conservation laws are not recovered. This is expected
since the general diffeomorphism invariance is broken by the unimodular
condition. But, it can be imposed as a new choice. In fact, if the relation
${T^{\mu\nu}}_{;\mu}=0$ is imposed, we obtain the following expression:

$\displaystyle R^{;\nu}=-8\pi GT^{;\nu}.$ (14)

which can be integrated, leading to,

$\displaystyle R=8\pi GT-\Lambda,$ (15)

where $\Lambda$ is an integration constant which is identified with the
cosmological term. The choice that the energy-momentum tensor conserves
separately leads to the GR equation with a cosmological term, that is,
equations, (2,4). On the other hand, if the conservation of the energy-
momentum tensor is not imposed, we have the field equations given (12) with
the generalized conservation law (13). It is these last set of equations that
we want to study here, considering the evolution of gravitational waves in a
cosmological context.

##  III A Cosmological model

Our next step in analyzing the nonconservative unimodular gravity theory is to
explore the evolution of gravitational waves in the cosmological context.
Cosmological models in unimodular gravity with a conserved energy-momentum
tensor has been discussed, for example, in Ref. alvarez . The present work
will be concentrate in this specific problem which may already give some
insight on the consequences of using equations (12) and (13). In these
equations we introduce the flat metric, with the lapse function given by
$N=1$, a possibility allowed as discussed in the previous section. Hence, the
line element is given by,

$\displaystyle ds^{2}=dt^{2}-a(t)^{2}(dx^{2}+dy^{2}+dz^{2}).$ (16)

What is remarkable is that the time-time and space-space components of (12)
lead to the same equation:

$\displaystyle\dot{H}=-4\pi G(\rho+p),$ (17)

where we have noted $H=\dot{a}/a$. It is worth noting that the above equation
implies that the vacuum case corresponds to the de Sitter solution in contrast
to the GR case where it corresponds to the Minkowski solution alvarez . This
fact indicates that the system of equations is underdetermined.

But we still have the conservation equation (13), which we can explore in
order to a complete set of self-consistent equations. The conservation laws
result in the following equation:

$\displaystyle\ddot{H}+4H\dot{H}$ $\displaystyle=$ $\displaystyle-4\pi
G[\dot{\rho}+\dot{p}+4H(\rho+p)],$ (18)

But both equations (17) and (18) surprisingly have the same content: inserting
(17) into (18) we obtain $0=0$. Even with the use of the generalized
conservation law, the system of equations remain underdetermined.

At this stage, two observations are in order: as already stated, the resulting
system of equations in the cosmological context (with isotropy and
homogeneity) is underdetermined, since there are two functions, $\rho$ and
$H$, for just one equation, remembering that the pressure is connected to
density by an equation of state; moreover, equations (17) and (18) are
sensitive only to the combination $\rho+p$, which is related to the enthalpy
of the system, and does not depend on the equation of state provided it is
barotropic, $p=p(\rho)$.

Defining $\bar{\rho}=\rho+p$, the equations (17,18) can be written as,

$\displaystyle\dot{H}$ $\displaystyle=$ $\displaystyle-4\pi G\bar{\rho},$ (19)
$\displaystyle\ddot{H}+4H\dot{H}$ $\displaystyle=$ $\displaystyle-4\pi
G(\dot{\bar{\rho}}+4H\bar{\rho}),$ (20)

simplifying the notation.

An interesting aspect is that the usual radiative solution of GR

$\displaystyle H=\frac{1}{2t},\quad\bar{\rho}=\bar{\rho}_{0}a^{-4},$ (21)

is also solution for (19) and (20) irrespective of the pressure $p$, except
for the case the pressure represents the vacuum energy or, alternatively, a
cosmological term: $p\neq-\rho$.

The fact that the radiative solution is also a solution of the nonconservative
unimodular equations may be related to the traceless property of the field
equations. Remark, however, that these field equations are traceless but not
conformal invariant. For $p=-\rho$ we find the usual de Sitter solution,
$a\propto e^{\kappa t}$, $\kappa$ a constant (positive or negative). Is there
any other solution? Since the system is underdetermined, we can not obtain any
other solution unless some additional hypothesis is introduced. We will later
provide more comments on this point.

To obtain a specific model, and taking into account the properties discussed
above, we can impose that both sides of (20) conserves separately. This leads
to,

$\displaystyle\ddot{H}+4H\dot{H}=0,$ (22)
$\displaystyle\dot{\bar{\rho}}+4H\bar{\rho}=0.$ (23)

The equation (23) has a simple solution:

$\displaystyle\bar{\rho}=\bar{\rho}_{0}a^{-4}.$ (24)

This is typical of a radiative fluid. However, we remember that
$\bar{\rho}=(\rho+p)$, hence the radiative behavior is independent of the
fluid considered. This is reflection of the traceless character of the field
equations and the nonconservation of the energy-momentum tensor (otherwise we
go back to GR structure).

Besides the behavior for any fluid, which is always similar to a radiative
fluid, independent of the corresponding pressure, there is another difference.
In general relativity, the equations for a radiative fluid are given by,

$\displaystyle H^{2}$ $\displaystyle=$ $\displaystyle\frac{8\pi G}{3}\rho,$
(25) $\displaystyle 2\dot{H}+3H^{2}$ $\displaystyle=$
$\displaystyle-\frac{8\pi G}{3}\rho.$ (26)

These equations lead to,

$\displaystyle\dot{H}+2H^{2}=0.$ (27)

We notice that equation (22) can be written as

$\displaystyle\frac{d}{dt}\biggr{(}\dot{H}+2H^{2}\biggl{)}=0.$ (28)

This is equivalent to the ansatz used in Ref. trace , $R=$ constant which, in
a similar context, has been introduced to complete the set of equations. The
GR case corresponds to set the constant equal to zero.

Following the reasoning exposed above, in the unimodular cosmology, without a
separated conservation of the energy-momentum tensor, the equation for the
Hubble function is given by,

$\displaystyle\dot{H}+2H^{2}=\frac{2}{3}\Lambda_{\rm U},\quad\Lambda_{\rm
U}=\mbox{constant}.$ (29)

The integration constant, which we have called $\Lambda_{\rm U}$ to keep the
contact with the cosmological term, makes the unimodular cosmological scenario
similar to the GR radiative model in presence of a cosmological constant, even
if no cosmological constant was present from the beginning. If fact an
integration constant similar to the cosmological constant is hidden in the
structure of unimodular gravity.

From (29) we have three possibilities.

$\displaystyle\Lambda_{\rm U}<0\quad$ $\displaystyle\rightarrow$
$\displaystyle\quad a=a_{0}\sin^{1/2}\sqrt{-\frac{4\Lambda_{\rm U}}{3}}t,$
(30) $\displaystyle\Lambda_{\rm U}=0\quad$ $\displaystyle\rightarrow$
$\displaystyle\quad a=a_{0}t^{1/2},$ (31) $\displaystyle\Lambda_{\rm
U}>0\quad$ $\displaystyle\rightarrow$ $\displaystyle\quad
a=a_{0}\sinh^{1/2}\sqrt{\frac{4\Lambda_{\rm U}}{3}}t.$ (32)

These are essentially the same solutions found in Ref. trace . The case
$\Lambda_{\rm U}=0$ is identical to the GR radiative model. If $\Lambda_{\rm
U}\neq 0$ we obtain formally the cosmological solutions in GR for a mixture of
radiation and cosmological constant. The solutions corresponding to
$\Lambda_{\rm U}\neq 0$ could also be expressed in terms of $\cos$ and $\cosh$
functions, which represent a nonsingular universe. But these possibilities
would imply negative energy density $\bar{\rho}$, which mounts to a violation
of the null energy condition since $\bar{\rho}$ leads to $\rho+p<0$, and for
this reason we avoid this possibility.

For all three values of $\Lambda$ the behavior for $t\rightarrow 0$ are
similar, and coincides with the flat radiative case. Of particular interest is
the case $\Lambda_{\rm U}>0$: it interpolates the initial radiative phase and
a de Sitter phase. In the usual GR context such possibility is generally not
explored since structure formation requires a matter dominated phase ($p=0$),
which would be here absent. But such restriction may be circumvented in the
nonconservative unimodular scenario as it is discussed in marcelo . The case
$\Lambda_{\rm U}<0$ corresponds to an interpolation between a radiative phase
and an anti-de Sitter phase which in practise is not interesting from the
observational point of view.

##  IV Gravitational waves

We will perform now the perturbative computation of the tensorial modes which
represent the gravitational wave phenomena. The background gravitational waves
in an expanding universe may be detected in future experiments. We address the
reader to some references on this important topic, see Refs. grisha ; shapiroo
and references therein.

At linear level, the computation of tensorial modes can be made by considering
fluctuations around a given background solution, such that

$\displaystyle g_{\mu\nu}=g_{\mu\nu}^{B}+h_{\mu\nu},$ (33)

where the superscript $B$ indicates the background metric and
$|h_{\mu\nu}|<<|g_{\mu\nu}^{B}|$ such that we can use the linear
approximation.

The tensorial modes are represented by the traceless, divergence-free spatial
components of $h_{\mu\nu}$ that behaves as pure tensorial quantity on the
three-dimensional spatial section. That is, the components $h_{ij}$ are such
that $h_{kk}=\partial_{k}h_{ki}=0$. The tensorial modes are invariant by any
coordinate transformation. In this sense, in their study, we can impose any
coordinate condition or use the gauge-invariant formalism. For simplicity we
use the synchronous coordinate condition, $h_{\mu 0}=0$. The perturbation of
the unimodular condition implies,

$\displaystyle h_{kk}=0,$ (34)

which is already encoded in the condition to have pure tensorial modes.

Using the previous relations, i.e., the perturbation of the Ricci components,
let us now perturb the Ricci scalar and the energy-momentum tensor. We
restrict ourselves only to the tensorial modes. This implies that the only non
zero components are the ones with the form wein-b ,

$\displaystyle\delta R_{ij}$ $\displaystyle=$
$\displaystyle-\frac{\ddot{h}_{ij}}{2}+\frac{H}{2}\dot{h}_{ij}-2H^{2}h_{ij},$
(35) $\displaystyle\delta T_{ij}$ $\displaystyle=$ $\displaystyle-ph_{ij}.$
(36)

With the help of the background equations and after perform a Fourier
decomposition, we find that the gravitational radiative modes obey the
equation,

$\displaystyle\ddot{h}_{ij}-H\dot{h}_{ij}-2(\dot{H}+H^{2})h_{ij}+\frac{k^{2}}{a^{2}}h_{ij}=0.$
(37)

This is exactly the equation for gravitational waves in GR with a perfect
fluid, see Ref. wein-b . This is by itself a surprising feature taking into
account the many differences between GR and the nonconservative unimodular
gravity. However, the evolution of gravitational waves in the above described
unimodular cosmological scenario and in the standard cosmological scenario
should be very different: the expressions for the background functions $H$ and
$\dot{H}$ are not the same, leading to special signatures in the evolution of
gravitational waves.

Let us return for the moment to equation (37) and review some of its property
in the case the universe is dominated by radiation and a cosmological constant
$\Lambda$. An analytic solution can be obtained when $\Lambda_{\rm U}=0$. In
this case, writing the scale factor in terms of the conformal time $\eta$
defined as $dt=ad\eta$, we obtain $a=a_{0}\eta$, and the equation becomes,

$\displaystyle
h^{\prime\prime}-2\frac{h^{\prime}}{\eta}+\biggr{\\{}k^{2}+\frac{2}{\eta^{2}}\biggl{\\}}h=0,$
(38)

where we have, for simplicity, ignored the polarization indices. This equation
can be solved analytically for an equation of state $p=\omega\rho$. The
solution reads grishagw

$\displaystyle
h=\eta^{(5+3\omega)/[2(1+3\omega)]}\biggr{\\{}A_{1}J_{\nu}(k\eta)+A_{2}J_{-\nu}(k\eta)\biggl{\\}}.$
(39)

where $A_{1,2}$ are constants and $J_{\nu}(x)$ is the Bessel function of order
$\nu$, with $\nu=3(1-\omega)/[2(1+3\omega)]$.

For a radiative universe ($\omega=1/3$), the solution is

$\displaystyle
h=\eta^{3/2}\biggr{\\{}A_{1}J_{1/2}(k\eta)+A_{2}J_{-1/2}(k\eta)\biggl{\\}}.$
(40)

The solution here is the same as in a cosmological model with only radiation
in GR. Remember, however, that in the unimodular model the solution is valid
irrespective of the choice made for $p$.

The solutions (40) have two asymptotic behavior:

$\displaystyle\eta\rightarrow 0\quad$ $\displaystyle\Rightarrow$
$\displaystyle\quad h\propto(A_{1}\eta^{3}+A_{2}\eta),$ (41)
$\displaystyle\eta\rightarrow\infty\quad$ $\displaystyle\Rightarrow$
$\displaystyle\quad h\propto\eta\cos(k\eta+\delta),$ (42)

where $\delta$ is a phase. Hence, initially the solutions exhibit growing
modes which for large values of time becomes a growing oscillatory mode.

We remark en passant that for the case of the de Sitter solution ($\omega=-1$,
$\nu=3/2$), the conformal time range is $-\infty<\eta<0$. Initially there are
growing oscillations that asymptotically become either growing or decreasing
modes, with no oscillations.

For the other possible solutions for the background in unimodular gravity
(with $\Lambda_{\rm U}>0$ or $\Lambda_{\rm U}<0$), it seems not possible to
obtain an closed expression for $h$. However, we can integrate numerically.
First we do it in a qualitative way rewriting equation (37) for the scale
factor. The result is,

$\displaystyle
h^{\prime\prime}+\frac{\dot{H}}{H^{2}}\frac{h^{\prime}}{a}+\biggr{\\{}\frac{k^{2}}{H^{2}a^{4}}-2\biggr{[}\frac{\dot{H}}{H^{2}}+1\biggl{]}\frac{1}{a^{2}}\biggl{\\}}h=0.$
(43)

Now we need the expressions for $H$ and $\dot{H}$ in terms of the scale
factor. In order to do so, we use (17), which already provides $\dot{H}$ in
terms of $a$ since we use that $\bar{\rho}\propto a^{-4}$. Then we multiply
that equation by $H$ and integrate. The resulting set of equations are,

$\displaystyle\frac{\dot{H}}{H_{0}^{2}}$ $\displaystyle=$
$\displaystyle-\frac{3}{2}\frac{\bar{\Omega}_{0}}{a^{4}},$ (44)
$\displaystyle\frac{H^{2}}{H_{0}^{2}}$ $\displaystyle=$
$\displaystyle\frac{3}{4}\frac{\bar{\Omega}_{0}}{a^{4}}+1-\frac{3}{4}\bar{\Omega}_{0},$
(45)

where we have defined,

$\displaystyle\bar{\Omega}_{0}=\frac{8\pi G\bar{\rho}_{0}}{3H_{0}^{2}},$ (46)

and we have fixed $a(t=t_{0})=1$, where $t_{0}$ is the present time. Remark
that the density parameter (46) is perfectly logical in a GR cosmology. But in
unimodular gravity this definition may be seen as a choice since $\bar{\rho}$
contains also the pressure.

We must make now a more detailed and realistic comparison with the standard
$\Lambda$CDM model in GR. This is the object of the next section.

##  V Comparing unimodular cosmology with the standard cosmological model

Here, we will make a more detailed comparison of the unimodular gravity the
evolution of the gravitational wave in the standard $\Lambda$CDM model.

In the $\Lambda$CDM model the matter content is given by radiation,
pressureless matter and the cosmological constant. In this case, the Friedmann
equations read,

$\displaystyle\frac{H^{2}}{H^{2}_{0}}$ $\displaystyle=$
$\displaystyle\frac{\Omega_{r0}}{a^{4}}+\frac{\Omega_{m0}}{a^{3}}+\Omega_{\Lambda},$
(47) $\displaystyle\frac{\dot{H}}{H_{0}^{2}}$ $\displaystyle=$
$\displaystyle-\frac{3}{2}\frac{\Omega_{m0}}{a^{3}}-2\frac{\Omega_{r0}}{a^{4}},$
(48)

where the subscript $m$ indicates matter, the subscript $r$ refers to
radiation and the density parameter is defined as before. The subscript $0$
indicates today’s values of the parameters. The matter component scales as
$a^{-3}$ while the radiation component scales as $a^{-4}$. The constraint
relation is now,

$\displaystyle 1=\Omega_{r0}+\Omega_{m0}+\Omega_{\Lambda}.$ (49)

Now, we go back to discuss the features of the nonconservative unimodular
cosmological model. The main point, at this stage, is what we interpret as the
matter parameter. The equations in unimodular gravity are sensitive only to
the combination $\rho+p$. Moreover, the system of equations is
underdetermined, hence an ansatz must be introduced. Our choice exposed above
leads to expressions that are similar to the radiative case in GR with a
cosmological constant. This property lead us to define,

$\displaystyle\bar{\rho}=\rho+p=\frac{\bar{\rho}_{0}}{a^{4}},$ (50)

with $\bar{\rho}_{0}$ indicating the value of the redefined density
$\bar{\rho}$ today. Using the definitions for the density parameter already
shown in the previous section, we obtain equation (29) with solutions (30-32).
The factor $3/4$ in these expressions is a consequence of our choices but
allows us to represent the background expansion in a familiar form. Even if
the the unimodular equations are sensible only to the combination $\rho+p$, if
we fix $p=\rho/3$ and using the same definitions for the density parameter, we
obtain exactly the equations for a cosmology with radiation and a cosmological
constant, as we could expect. But we will keep ourselves within the context of
the unimodular structure with a generalized conservation law.

With these definitions in mind, we will plot the cosmological evolution of
gravitational waves in two scenarios: the unimodular model and the concordance
$\Lambda$CDM model. We remark that in both cases the evolution of
gravitational waves is dictated by equation (43), differing only on the
background functions $H$ and $\dot{H}$.

First we fix $\bar{\Omega}_{0}=0.05$, corresponding to the total amount of
baryonic matter, for the unimodular. For the $\Lambda$CDM model we fix
$\Omega_{m0}=0.3$ and $\Omega_{r0}=5\times 10^{-5}$ for the standard model.
The result is displayed in Fig. 1. In Fig. 2 we repeat the computation, but
imposing now $\bar{\Omega}_{0}=0.95$, keeping the same values for the
$\Lambda$CDM model, in order to stress the difference between the models. We
remark that the unimodular case present a higher amplification of the
gravitational waves compared with the standard cosmological model for any
value of the parameter $\Omega_{0}$. Also the frequency of the waves differ
significantly. This may have clear observational signatures.

Figure 1: Evolution of gravitational waves in unimodular gravity (continuous
line) with $\bar{\Omega}_{0}=0.05$ and in the $\Lambda$ CDM model (dashed
line) with $\Omega_{m0}=0.3$ and $\Omega_{r0}=5\times 10^{-5}$. The initial
conditions are the same. Figure 2: Evolution of gravitational waves in
unimodular gravity (continuous line) with $\bar{\Omega}_{0}=0.95$ and in the
$\Lambda CDM$ model (dashed line) with $\Omega_{m0}=0.3$ and
$\Omega_{r0}=5\times 10^{-5}$. The initial conditions are the same.

##  VI Discussion and conclusions

Unimodular gravity is a theory of gravity almost as old as GR. It is
characterized by the condition $g=$ constant. One of the attractive aspects of
modular gravity is that by, imposing the usual conservation law for the
energy-momentum tensor, the RG equations are recovered with an integration
constant that can be identified with the cosmological term $\Lambda$. This
fact brings a new perspective to the cosmological constant problem wein-a . At
quantum level, unimodular gravity has also many interesting features and
discussed in quantum1 .

In this paper, we investigate one variant of unimodular gravity not very
explored in the literature. As observed above, the conservation of the energy-
momentum tensor is a free choice within the structure of theory, since the
general invariance on diffeomorphism is lost when we fix the determinant of
the metric. The consequences of not imposing the usual conservation laws were
stressed here. The first one is that, in the cosmological framework, the
equations for the scale factor are sensitive only to the combination $\rho+p$,
irrespective of the fluid. Moreover, the equations are underdetermined, and
they can not be solved without an ansatz.

We have showed that the typical solution of radiation with (and without) a
cosmological constant is consistently also a solution of the nonconservative
unimodular gravity. As a remark, from this result, we can speculate that the
cosmological constant is always hidden in the unimodular gravity, irrespective
of the conservation of energy-momentum tensor or not.

In order to distinguish the unimodular cosmological model from the GR
cosmological models, we have decided to investigate the evolution of
gravitational waves in these different scenarios. The first obstacle is to
identify the density parameters in unimodular gravity since all equations are
only sensitive to the combination $\rho+p$, the entalphy of the system. This
quantity scales as $a^{-4}$ as a legitime radiative field. We have identified
this combination as the matter content of the model, and the density parameter
has been defined in order to keep close contact with the similar definitions
in the standard cosmological model.

We have then studied the evolution of gravitational waves in the different
cosmological scenarios emerging from the unimodular gravity in comparison with
the Standard Cosmological Model, with radiation, matter and cosmological
constant. The main feature is that unimodular gravity as well as the radiation
plus cosmological constant scenario in GR provides a stronger amplification of
gravitational waves amplitudes compared the $\Lambda$CDM model. Remark that if
we choose a radiative equation of state, $p=\rho/3$, the unimodular scenario
becomes identical to the radiative case with a cosmological constant in GR
given the definition for the density parameter chosen.

Hence, it is possible in principle to distinguish the unimodular cosmological
scenario from the $\Lambda$CDM model through the amplitude and the shape
(frequency) of gravitational waves. However, in case future observations
provide gravitational wave signals in agreement with the unimodular
prediction, what could distinguish it from the radiative model with
cosmological constant, since the features are very similar at least in what
concerns the evolution of gravitational wave? To try to answer this question,
we must remember that structure formation in a cosmological model based in GR
can not take place without a matter dominated phase, and we come back to the
$\Lambda$CDM model. But, structure formation may take place in the unimodular
scenario exhibited here, in spite of its radiative character, as some
preliminary results indicate marcelo .

The results here reported are somehow qualitative, indicating the main
characteristic features of the evolution of gravitational waves in unimodular
gravity. A more detailed analysis of the observational signatures is necessary
to clearly distinguish the different scenarios. Of course, the analysis of
structure formation, which is currently under study, may be another powerful
way to investigate the viability of nonconservative version of unimodular
cosmology.

However, the unimodular scenario discussed here seems to be incomplete because
it is necessary an additional condition to solve the equations. In the present
work we have imposed the same condition as in reference trace . Perhaps the
nonconservative unimodular gravity must be complemented by some version of the
holographic principle. We intend to perform in the future.

It would be interesting also to compare in details the results here obtained
with those coming, for example, from the mimetic gravity.

## Acknowledgments

We thank to CNPq, CAPES, FAPES and PROPP/UFOP for financial support.

## References

  * (1) S. Weinberg, Rev. Mod. Phys 61, 1(1989).
  * (2) J.J. Lopez-Villarejo, JCAP11, 002(2011).
  * (3) A. Eichhorn, Class. Quant. Grav. 30 (2013), 115016 doi:10.1088/0264-9381/30/11/115016 [arXiv:1301.0879 [gr-qc]]; A. Padilla and I. D. Saltas, Eur. Phys. J. C 75 (2015) no.11, 561 doi:10.1140/epjc/s10052-015-3767-0 [arXiv:1409.3573 [gr-qc]]; I. D. Saltas, Phys. Rev. D 90 (2014) no.12, 124052 doi:10.1103/PhysRevD.90.124052 [arXiv:1410.6163 [hep-th]]; E. Álvarez, S. González-Martín, M. Herrero-Valea and C. P. Martín, JHEP 08 (2015), 078 doi:10.1007/JHEP08(2015)078 [arXiv:1505.01995 [hep-th]]; R. Bufalo, M. Oksanen and A. Tureanu, Eur. Phys. J. C 75 (2015) no.10, 477 doi:10.1140/epjc/s10052-015-3683-3 [arXiv:1505.04978 [hep-th]]; R. Percacci, Found. Phys. 48 (2018) no.10, 1364-1379 doi:10.1007/s10701-018-0189-5 [arXiv:1712.09903 [gr-qc]]; G. P. de Brito and A. D. Pereira, JHEP 09 (2020), 196 doi:10.1007/JHEP09(2020)196 [arXiv:2007.05589 [hep-th]]; G. P. de Brito, O. Melichev, R. Percacci and A. D. Pereira, [arXiv:2105.13886 [gr-qc]].
  * (4) M. Herrero-Valea, JHEP 12 (2018), 106 doi:10.1007/JHEP12(2018)106 [arXiv:1806.01869 [hep-th]].
  * (5) G.F.R. Ellis, Gen. Rel. Grav. 46, 1619(2014).
  * (6) A. H. Chamseddine and V. Mukhanov, JHEP 13, 135(2013).
  * (7) A.H. Chamseddine, V. Mukhanov and A. Vikman, JCAP 06, 17(2014).
  * (8) M. Alvarenga, J.C. Fabris, M. Hamani-Daouda and H. Velten, work in progress.
  * (9) Cai-xia Gao, Robert H. Brandenberger, Yi-fu Cai and Pi-sin Chen, JCAP, 09, 021(2014).
  * (10) R.M. Wald, General Relativity, Chicago university press, Chicago (1984).
  * (11) E. Alvarez and J. Anero, Unimodular cosmological models, arXiv: 2109.08077
  * (12) M. Daouda, J.C. Fabris, A.M. Oliveira, F. Smirnov and H.E.S. Velten, Int. J. Mod. Phys. D 28, 1950175(2019).
  * (13) L.P. Grishchuk, Phys. Usp. 48,1235(2005).
  * (14) B. Li and P.R. Shapiro, JCAP 10, 024(2021).
  * (15) S. Weinberg, Gravitation and cosmology, Wiley, New York (1972).
  * (16) L.P. Phys. Rev. D48, 3513(1993).


Maureen de Seyssel1,2, Marvin Lavechin1,5,Hadrien Titeux1, Arthur Thomas†,
Gwendal Virlet4†, Andrea Santos Revilla†, Guillaume Wisniewski2, Bogdan
Ludusan3, Emmanuel Dupoux1,5

# ProsAudit, a prosodic benchmark for self-supervised speech models

###### Abstract

We present ProsAudit, a benchmark in English to assess structural prosodic
knowledge in self-supervised learning (SSL) speech models. It consists of two
subtasks, their corresponding metrics, and an evaluation dataset. In the
protosyntax task, the model must correctly identify strong versus weak
prosodic boundaries. In the lexical task, the model needs to correctly
distinguish between pauses inserted between words and within words. We also
provide human evaluation scores on this benchmark. We evaluated a series of
SSL models and found that they were all able to perform above chance on both
tasks, even when evaluated on an unseen language. However, non-native models
performed significantly worse than native ones on the lexical task,
highlighting the importance of lexical knowledge in this task. We also found a
clear effect of size with models trained on more data performing better in the
two subtasks.

Index Terms: prosody, speech representation, self-supervised learning, human
evaluation

##  1 Introduction

In recent years, self-supervised learning (SSL) speech models such as Wav2Vec
[1], CPC [2, 3], HuBert [4] have made groundbreaking advancements, while
removing the need for labeled data as they use information extracted from the
input raw audio itself. Multiple benchmarks and metrics have been since
developed to test (and exhibit) the linguistic knowledge of such models at
different levels. For instance, the Zero-Resource speech challenge [5, 6, 7]
offers zero-shot evaluation metrics at the phonetic, lexical, syntactic and
semantic levels and SUPERB [8] allows downstream evaluation on speech
processing tasks, including paralinguistics and speaker-related tasks.

An important aspect of language that has received little to no attention in
SSL models is prosody. As a result, we still know little about SSL encoding of
structural prosodic knowledge. Yet, through its key components: rhythm,
stress, and intonation, prosody plays a significant role in language
processing [9, 10], interfacing with other linguistic levels (e.g., lexical,
syntactic levels), while also carrying paralinguistic information (e.g.,
emotion) (for a literature review, see [9] and [10]). Recently, [11] proposed
to explicitly implement prosodic knowledge into SSL models by forcing
prediction of pitch and duration within the models, but only human evaluation
on downstream tasks was used as an indirect cure of prosodic encoding in the
models. Besides, [12] proposed a benchmark to test pragmatics aspects of
prosody of the models using downstream tasks. Yet, there is currently no zero-
shot metrics which allows systematic evaluation of prosody in such models.

In this paper, we fill this gap by proposing an evaluation benchmark,
ProsAudit, which assesses SSL speech models' ability to learn prosodic
information at the structural level. By such, we refer to how prosody
contributes to the organisation of speech by marking the boundaries between
words, phrases, and sentences. Our benchmark, in English, consists of two
subtasks. The protosyntax task tests the model's ability to identify strong
versus weak prosodic boundaries (e.g., see [13] for an evaluation of human
performance). The lexical task tests the model's ability to distinguish
between pauses inserted between and within words. Being a proxy for detecting
word boundary versus word internal, this task requires some lexical knowledge.
Crucially, we also provide results on these two tasks carried out on human
evaluation. We make our benchmark available online, and present some baseline
results. Finally, we conduct further analysis on factors like input quantity
and nativeness, putting the models in perspective with prosodic learning in
humans.

##  2 Methods

Table 1: Examples of stimuli in the English benchmark, for the ProtoSyntax and Lexical tasks. Numbers in subscript correspond to prosodic break tiers in ToBI format. task | condition | stimuli
---|---|---
protosyntax | natural |  4She1 went1 to1 jail4 $<$PAUSE$>$ for1 what1 appeared1 to1 be1 a1 murder4
protosyntax | unnatural |  4She1 went1 to1 jail4 for1 what1 appeared1 to1 $<$PAUSE$>$ be1 a1 murder4
lexical | natural |  4She1 went1 $<$PAUSE$>$ to1 jail4 for1 what1 appeared1 to1 be1 a1 murder4
lexical | unnatural |  4She1 went1 to1 jail4 for1 what1 a-$<$PAUSE$>$ -ppeared1 to1 be1 a1 murder4

###  2.1 ProsAudit benchmark

We created prosodic benchmarks in English, composed of two tasks which each
focus on different aspects of structural prosody: the lexical task and the
protosyntax task. These tasks are designed to evaluate the models'
understanding of prosody by presenting them with pairs of stimuli that only
differ in the placement of a pause.

####  2.1.1 Material and data preprocessing

We used the Boston University Radio News Corpus (BU) [14] dataset to create
the evaluation set, as it includes word and phone-level transcriptions along
with prosodic hierarchy annotations based on the American English ToBI system
[15]. The dataset is a collection of professionally-read news stories.

We selected segments based on the criteria outlined in [13]. To qualify, a
segment had to start and end with an intonation phrase (IP) boundary (ToBI
level 4) and contain one internal prosodic boundary, which could be an IP
boundary or an intermediate boundary (ToBI level 3). Additionally, the
qualifying segments had to meet specific criteria: a minimum and maximum
duration of 2 and 5 seconds; the internal prosodic boundary could not be
between the two first or two last syllables of the utterances, and a pause
should be annotated wherever a prosodic IP was annotated111This is not
systematic as IP can be present without the speaker marking a pause. The next
step consisted in automatically deleting all existent annotated pauses from
the stimuli, applying some crossfading (10ms on each side of the pause) to
prevent abrupt cut-off or jump in the audio.

####  2.1.2 Creating the protosyntax and lexical tasks

In the protosyntax task, one stimulus has a pause placed at a ``natural"
location : a prosodic phrase boundary (ToBI levels 3 or 4). In contrast, the
other stimulus has the pause placed at an ``unnatural" location where there
are no higher level prosodic breaks (ToBI levels 1 or 2). This task aims to
assess the models' understanding of structural prosody at the sentence level.
In the lexical task, the prosodic boundary is present at a word boundary in
the natural condition (levels 1 or 2) and within a word in the unnatural
condition. Because the differences between these two conditions are less
marked prosodically, lexical knowledge should be required on top of prosodic
knowledge to perform well in this task. Examples of both tasks are presented
Table 1.

While there is only one possibility for where the pause can be in the natural
condition, multiple stimuli can be created for each pair in the unnatural
condition. Therefore, we included constraints on the position of the break
from the start and end of the audio regarding the number of syllables and
seconds. We sampled the final stimuli by implementing similarity losses for
these constraints, aiming for the smallest divergence between the
distributions of the natural and unnatural conditions. Once the stimuli pairs
and break positions were defined, we inserted a 400ms pause with crossfading
at the chosen position, resulting in a total of 5,234 pairs in the protosyntax
task and 5,178 pairs in the lexical task. Since the pause is artificially
inserted in both unnatural and natural conditions, the duration of the two
stimuli is the same in both conditions, with the only difference being the
location of the pause within the stimuli. Finally, we randomly sampled about
10% of the pairs to create a dev set, resulting in 2,355 (262) pairs in the
protosyntax task for the test (dev) set and 2,330 (259) pairs in the lexical
task for the test (dev) set.

####  2.1.3 Metric

We employed, in both subtasks, the same metric as the sWuggy and sBlimp
metrics proposed in [6] : we compute, for a pair of stimuli, the probability
of the model evaluated (or pseudo-probability) to generate these stimuli. If
the probability of the “natural” stimuli is higher than the unnatural stimuli,
we give a score of 1 for this pair, otherwise a score of 0. The final score
corresponds to the average score for all dev or test pairs in the given
subtask (see [6] for more details).

###  2.2 Baselines

We evaluated several self-supervised learning models of speech on the
protosyntax and lexical tasks, as well as a human baseline.

####  2.2.1 GSLM and pGSLM baselines

We evaluated our models against the prosody-aware models presented in [11],
which we refer to as pGSLM models. These models are an extension of the “GSLM”
models, presented in an earlier study [16], and that we also evaluated. All of
these models are trained on a “clean” 6k hours sub-sample of the Libri-Light
dataset [17], which is made of English audiobooks. The GSLM models have three
main components: an acoustic model (HuBert), a quantizer, and a language model
(transformer). Here, we present two versions of these GSLM models, the
standard version and the “deduplicated” version, where the units output by the
quantizer are deduplicated. The pGSLM models build upon the GSLM models by
adding tasks for the language model, which predicts the fundamental frequency
(f0) and duration of the following frames, in addition to the frame's content.
Because the duration is predicted, the information is also removed from the
deduplicated units.

The f0 and duration prediction tasks should enable the model to learn the
rhythm and intonation of the speech, which are important cues for
understanding the meaning of speech. Hence, this modification allows the pGSLM
models to incorporate prosodic information into their predictions. By
incorporating this information, the pGSLM models may be able to produce more
natural-sounding speech and make more accurate predictions. In this paper, we
consider four versions of these pGSLM models, trained on continuous (cont.) or
discrete (disc.) input, and with or without a prosodic shift in the prediction
(see [11] for more details). However, we only compute the pseudo-probability
based on the original LM token prediction, similarly to what is done in [11].

####  2.2.2 STELA baselines

The tasks were evaluated on English and French models from [18], which we
refer to as STELA models. By also using models trained in an unseen language
(French), we can test the effect of nativeness on structural prosody. These
models, trained on English and French audiobooks, use a Contrastive Predictive
Coding objective [2, 3] for the first acoustic model, which generates
continuous representations. These representations are then quantized using
k-means, and a language model (LSTM) is trained on the resulting units.
Additionally, these models are trained on varying amounts of data, allowing
for the creation of training size curves222To ensure comparable results,
models at all train sizes are overall trained on the same amount of data, with
more models being trained on the smallest train sizes. See [18] for more
details.. To ensure a fair comparison with other models, a ``deduplicated"
version of the 3,200 hour English model was also trained.

####  2.2.3 Human evaluation

Finally, we ran a human evaluation by presenting the same protosyntax and
lexical tasks in an online experiment. We used Mechanical Turk to recruit 389
participants for 790 sessions, which sessions were compensated $1.5 USD each.
In a session, the participant was presented with a series of 7 pairs from the
protosyntax and 7 pairs from the lexical task in a random order, along with
two examples at the beginning of the session, and four additional 4
controls333For a control pair, the pause is inserted at an intonation phrase
boundary (natural) and within a word (unnatural). specifically easier in order
to filter out participants who were not paying attention to the task. For each
pair, a webpage was presented to them, asking them to listen to two audios and
decide which of the two audio was the most natural (the condition order was
randomised), as well as how sure they were of their decision (slightly,
moderately, strongly). We also included two examples at the beginning of the
test, as well as 4 controls in order to filter out participants who were not
doing the task correctly. Participants were paid $1.5 USD per session. For the
analyses, we discarded all non-native English participants, as well as all
sessions where the participant did not correctly pass at least 5 out of the 6
examples + controls, resulting in a total of 255 participants (515 sessions).
Finally, we only included in our analysis stimuli pairs that were listened to
at least 5 times. We refer to these final stimuli as the “human subset”,
independent from the dev and test sets, which is composed of 521 pairs for the
protosyntax task and one of 510 pairs for the lexical task.

##  3 Results

###  3.1 Benchmark

Table 2: ProsAudit accuracy scores (%) for the different models.

|  | protosyntax | lexical
---|---|---|---
model | dataset | dev | test | dev | test
STELA | 3200h audiob. | 72.5 | 74.9 | 68.7 | 68.3
STELA deduplicated | 3200h audiob. | 58.0 | 58.5 | 48.7 | 46.7
GSLM | Librilight 6k | 58.8 | 58.1 | 53.3 | 54.1
GSLM deduplicated | Librilight 6k | 67.2 | 66.5 | 73.8 | 70.5
pGSLM - cont. | Librilight 6k | 65.7 | 66.8 | 73.8 | 71.5
pGSLM - cont. + shift | Librilight 6k | 69.1 | 66.8 | 74.9 | 71.9
pGSLM - disc. | Librilight 6k | 67.6 | 64.8 | 74.5 | 71.1
pGSLM - disc. + shift | Librilight 6k | 69.1 | 65.9 | 72.6 | 72.9

The benchmark (with both the dev and test sets) is available online444More
information along with url to the benchmark will be made available after the
double-blind review..

Scores on the protosyntax and lexical prosodic tasks evaluated on English
models are presented in Table 2. First, all models perform above chance in
both the protosyntax and lexical tasks, suggesting that all of these models
have some prosodic knowledge about the structure of sentences and words.

An interesting observation is that while the GSLM and pGSLM models perform
better on the lexical task, suggesting a better understanding of word
boundaries, the STELA models actually perform better on the protosyntax task,
indicating a stronger sensitivity to the prosodic structure of sentences. This
is surprising given that the pGSLM models are specifically trained to also
predict the duration and pitch of the next frames, in addition to the content
of the frame. However, it must be reminded here that the pseudo-probability
for the pGSLM model is only computed at the token level. A potential
improvement for future work would be considering the duration and pitch losses
when computing the pseudo-probability.

There is little difference in performance between the four pGSLM models,
although the continuous pGSLM with prosodic shift performs slightly better
overall. More surprisingly, the GSLM model with deduplicated units performs
similarly to the pGSLM models, even though the deduplication process removes
the duration information, which is an essential aspect of prosodic information
(while the GSLM models are also trained on deduplicated units, the duration
information is reinjected as a separate loss). This suggests that the lexical
and grammatical information present in the units may be sufficient for the
model to perform well on the two tasks, even without the prosodic information.
It is possible that the model learned to extract the relevant information from
the units even with the duration information removed.

Finally, while deduplicating the units in the GSLM model greatly helps in both
tasks, the opposite happens when deduplicating the units with the STELA model,
which yields much poorer results. While surprising at first, this could be
caused by multiple factors that vary between the two model types: the quantity
of data (there is nearly double the amount of data in the GSLM models) and the
architecture of the language model (an LSTM in the case of the STELA model and
a transformer in the GSLM ones). It would be interesting to study further the
impact of deduplication on other language evaluation tasks for these two types
of models.

Figure 1: ProsAudit accuracy scores (%) for the Protosyntax and Lexical
subtasks, on the STELA models, w.r.t. the training size.

###  3.2 Further analyses

To better understand the presence of structural prosody in SSL models, we
conducted a series of analyses using our ProsAudit benchmark.

Human/Machines comparison. Table 3 shows the scores of the human cohort on the
lexical and protosyntax tasks, compared to the best-scoring STELA and (p)GSLM
models on the same human subset555Computation of the correlation between human
and machine scores for the different items was not possible due to the high
variability and insufficient number of responses per items in the human
condition.. Humans perform better than chance on both tasks. The protosyntax
results (80.5%) are slightly lower than those presented in [13] (93.2%), where
participants were presented with a Japanese version of the task. However, this
can be explained by the fact that the stimuli in [13] were much more curated
and manually selected than the ones in ProsAudit, which compensates this by
having much more stimuli. Still, in the protosyntax task, humans perform
better than the models, acting as a topline. On the lexical task, both models
score higher than humans, who were less confident in their ratings than in the
protosyntax task. This may be due to the wording of the experiment, asking the
participants to choose the most natural sentence without any cues to focus on
the lexical aspect. Moreover, in this task, both conditions sound pretty
unnatural, as the ``correct" stimuli have a pause placed at a word boundary
with no prosodic break. This hypothesis is supported by additional analyses
indicating lower participant confidence in the lexical task compared to the
protosyntax task.

Table 3: ProsAudit accuracy scores (%) for the human evaluation and best
performing STELA and GSLM models, on the test and human subset.

| protosyntax | lexical
---|---|---
| test | human set | test | human set
humans native | - | 80.50 | - | 60.38
STELA | 74.86 | 73.32 | 68.28 | 71.18
pGSLM | 65.94 | 64.88 | 72.88 | 73.33

Effect of size. Results averaged over all STELA models from a same size of the
training dataset, both in English and French, are presented in Figure 1.
Focusing solely on the English models (solid line), we can note multiple
things. First, even models trained on a small amount of data (50 hours)
achieve above-chance performance in the prosodic tasks, indicating that
structural prosodic knowledge can be acquired from limited data. Additionally,
the results show a clear trend of improvement in performance as the size of
the training dataset increases, particularly for the lexical task, which
exhibits a logarithmic growth. However, in the case of the protosyntax task,
the study observes a plateau in the improvement of performance between 1,600
and 3,200 hours, potentially indicating a ceiling effect, meaning that beyond
this point, increasing the size of the dataset does not have a significant
impact on the model's performance.

Effect of nativeness. We analysed the performance of models trained on French
data when evaluated on English prosodic tasks, as depicted by the dotted line
in Figure 1. When considering only models trained on the largest amount of
data, it is found that there is a clear native advantage for English models,
achieving better performance than their French counterparts in both tasks.
This difference is more pronounced in the lexical task, where a strong lexical
knowledge of English is a key factor for success. A comparison of the overall
developmental curves reveals that while the performance of French models
improves in the protosyntax task with increasing training data, this is not
the case for the lexical task. The lack of improvement for French models in
this second task results in a widening performance gap between native and non-
native models. These findings, although unsurprising, indicate that non-native
models cannot acquire the necessary lexical knowledge to excel in this task
solely through additional training data.

##  4 Discussion & Conclusion

We introduced ProsAudit, a zero-shot benchmark for measuring the English
prosodic knowledge of speech SSL models, made of two subtasks. The protosyntax
task assesses the model's understanding of structural prosody at the syntactic
level. All the models we evaluated performed well above chance in this task,
indicating that this knowledge is embedded in speech SSL models. Besides,
models trained on another language perform relatively well in this task, which
is in line with findings in humans [13]. This suggests that some prosodic
knowledge can be universal [19]. However, the gap between native and non-
native models increases with more data, suggesting that some prosodic cues are
language-specific, indicating that the models are in line with findings from
the psycholinguistics literature [20].

The lexical task requires both hierarchical prosodic knowledge at the sentence
level and some lexical knowledge, as prosody is not always enough to
differentiate between word boundary and within word breaks. The GSLM models
(with and without prosody) performed better on this task than the protosyntax
one. This could be because they have some strong lexical knowledge of English
(these models perform relatively well on lexical metrics, see [16, 7]),
levelling up with their knowledge of prosody. Conversely, the STELA models do
not perform as well on this task as the protosyntax one, although they still
score way above chance, suggesting their lexical knowledge is not strong
enough to surpass the protosyntax task. Unsurprisingly, there is a much larger
native effect in this task, with non-native models performing only slightly
above chance, regardless of the size of the training set. In contrast, the
native models' performance is strongly correlated with data size. Further
research could examine how much lexical knowledge in a model correlates with
their performance on the ProsAudit lexical task.

We also found that the pGSLM models, despite supposedly encoding prosody-
specific features and yielding better mean opinion scores on downstream tasks
[11], score only slightly better on the protosyntax and lexical tasks than
their vanilla GSLM counterparts. Two things could explain these results.
First, we only compute the pseudo-probability taking into account the loss at
the token level (discarding losses at the pitch and duration levels), and
finding news ways to generate this pseudo-probability by taking into account
these two components could increase the models' scores. Second, the benchmark
we propose here only evaluates specific aspects of prosody at the structural
level, while other aspects of prosody as emotion, not evaluated in this
benchmark, could be more related to the metrics presented in [11].

To conclude, we propose for the first time a new zero-shot benchmark for
evaluating structural prosodic knowledge in speech models, along with a human
evaluation topline. We hope that it will inspire further research to enhance
the prosodic capabilities of SSL models. This is crucial, as research has
demonstrated that models with better prosodic understanding lead to more
advanced generative speech models (as seen in [11]). Additionally, the
benchmark is now made available online, providing a platform for continuous
improvement and comparison of models. In the future, we aim to develop this
benchmark in languages other than English, and to expand the benchmark to
subtypes targeting other aspects of prosody.

## References

  * [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [2] A. v. d. Oord, Y. Li, and O. Vinyals, ``Representation learning with contrastive predictive coding,'' _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [3] M. Riviere, A. Joulin, P.-E. Mazaré, and E. Dupoux, ``Unsupervised pretraining transfers well across languages,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7414–7418.
  * [4] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [5] E. Dunbar, X. N. Cao, J. Benjumea, J. Karadayi, M. Bernard, L. Besacier, X. Anguera, and E. Dupoux, ``The zero resource speech challenge 2017,'' in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 323–330.
  * [6] T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, A. Baevski, E. Dunbar, and E. Dupoux, ``The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,'' in _NeuRIPS Workshop on Self-Supervised Learning for Speech and Audio Processing_ , 2020.
  * [7] E. Dunbar, M. Bernard, N. Hamilakis, T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, and E. Dupoux, ``The Zero Resource Speech Challenge 2021: Spoken Language Modelling,'' in _Proc. Interspeech 2021_ , 2021, pp. 1574–1578.
  * [8] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``SUPERB: Speech Processing Universal PERformance Benchmark,'' in _Proc. Interspeech 2021_ , 2021, pp. 1194–1198.
  * [9] A. Cutler, D. Dahan, and W. Van Donselaar, ``Prosody in the comprehension of spoken language: A literature review,'' _Language and speech_ , vol. 40, no. 2, pp. 141–201, 1997.
  * [10] D. Dahan, ``Prosody and language comprehension,'' _Wiley Interdisciplinary Reviews: Cognitive Science_ , vol. 6, no. 5, pp. 441–452, 2015.
  * [11] E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Riviere, A. Mohamed, E. Dupoux _et al._ , ``Text-free prosody-aware generative spoken language modeling,'' in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022, pp. 8666–8681.
  * [12] G.-T. Lin, C.-L. Feng, W.-P. Huang, Y. Tseng, T.-H. Lin, C.-A. Li, H.-y. Lee, and N. G. Ward, ``On the utility of self-supervised models for prosody-related tasks,'' in _2022 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2023, pp. 1104–1111.
  * [13] B. Ludusan, M. Morii, Y. Minagawa, and E. Dupoux, ``The effect of different information sources on prosodic boundary perception,'' _JASA Express Letters_ , vol. 1, no. 11, p. 115203, 2021.
  * [14] M. Ostendorf, P. J. Price, and S. Shattuck-Hufnagel, ``The boston university radio news corpus,'' _Linguistic Data Consortium_ , pp. 1–19, 1995.
  * [15] K. E. Silverman, M. E. Beckman, J. F. Pitrelli, M. Ostendorf, C. W. Wightman, P. Price, J. B. Pierrehumbert, and J. Hirschberg, ``Tobi: A standard for labeling english prosody.'' in _ICSLP_ , vol. 2, 1992, pp. 867–870.
  * [16] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed _et al._ , ``On generative spoken language modeling from raw audio,'' _Transactions of the Association for Computational Linguistics_ , vol. 9, pp. 1336–1354, 2021.
  * [17] J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen _et al._ , ``Libri-light: A benchmark for asr with limited or no supervision,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7669–7673.
  * [18] M. Lavechin, M. de Seyssel, H. Titeux, H. Bredin, G. Wisniewski, A. Cristia, and E. Dupoux, ``Can statistical learning bootstrap early language acquisition? a modeling investigation,'' _PsyArXiv preprint PsyArXiv:rx94d_ , 2022.
  * [19] A. D. Endress and M. D. Hauser, ``Word segmentation with universal prosodic cues,'' _Cognitive psychology_ , vol. 61, no. 2, pp. 177–199, 2010.
  * [20] B. Höhle, R. Bijeljac-Babic, B. Herold, J. Weissenborn, and T. Nazzi, ``Language specific prosodic preferences during the first half year of life: Evidence from german and french infants,'' _Infant Behavior and Development_ , vol. 32, no. 3, pp. 262–274, 2009.