# Gravitational Lensing by Wormholes

Juan Manuel Tejeiro S  Eduard Alexis Larranaga R

###### Abstract

Natural wormholes and its astrophysical signatures have been sugested in
various oportunities. By applying the strong field limit of gravitational
lensing theory, we calculate the deflection angle and magnification curves
produced by Morris-Thorne wormholes in asimptotically flat space-times. The
results show that wormholes act like convergent lenses. Therefore, we show
that it is hard to distinguish them from black holes using the deflection’s
angle of the gravitational lens effect, in contrast with the results reported
by Cramer et.al. and Safanova et.al. However, we also show that it is
possible, in principle, distinguish them by the magnification curves, in
particular, by observing the position of the peak of the Einstein’s ring.

_Observatorio Astronomico Nacional. Universidad Nacional de Colombia_

##  1 Introduction

Wormholes are space-time regions with two mouths connected by a throat. Since
the mouths are not hidden by event horizons and there is no singularity
inside, wormholes permit the passage of massive particles or photons from one
side to the other. Morris and Thorne[1, 2] shown that wormholes require the
violation of the averaged null energy condition in order to satisfy the
Einstein field equations in the throat. Thus, matter in this region must exert
gravitational repulsion to make a stable configuration. The study of these
solutions lead to an interesting question: May some observable electromagnetic
signatures give the possibility of wormhole identification?

One of the most important applications of General Relativity is Gravitational
Lensing. Since matter inside the wormhole antigravitate, Cramer et. al. [5]
and Safanova et. al. [6] have studied the lensing effects of negative masses
on light rays from point sources in the background, and argumented that this
will be the effect produced by natural stellar-size wormholes. The study of
lensing effects show that positive masses acts like convergent lenses, while
the study of Safanova et. al. for negative masses show that these act
basically like divergent lenses. This means that the deflection angle is
negative (divergent). Therefore, light from a point source will be
concentrated on the border of an _umbra region_ , producing two intensity
enhancements: one before and one after the occultation event.

Recently, the scientific community started to look gravitational lensing from
a strong field perspective. This limit refers to the effect produced by a very
massive object and/or when the light rays that are affected pass very close to
the lens. Bozza [7] provided an analytic method to calculate the deflection
angle for any spherically symmetric spacetime in the strong field limit, and
one important result of this is that a light ray with a very small impact
parameter will wind one or several times around the lens before emerging,
producing an infinite set of _relativistic images_.

In this paper, we will use the method of Bozza to calculate a more realistic
lensing effect of Morris-Thorne wormholes, and it will permit us to show that
wormhole spacetimes present a _positive deflection angle_ , i.e. wormholes
does not act like divergent lenses in the general case. This means that exotic
matter needed to construct the solutions does not have a direct observable
signature for an outside observer.

In section II we give the metric of the studied wormhole solutions, section
III gives the essential knowledge about strong gravitational lensing as
described by Bozza and in section IV we calculate the deflection angle for the
presented wormholes. In section V we compare the obtained angles with the
Schwarzschilds deflection angle and finally, in section VI we put some
conclusions.

##  2 Morris-Thorne Wormholes

In the pionnering work of Morris and Thorne they describe a general class of
solution of Einstein equations representing wormholes. The conditions that
they impose in order to obtain the general form of the metric tensor include:

1\. The metric must be spherically symmetric and static (independient of time)

2\. The solution must have a throat joining two asymptotically flat regions of
spacetime

3\. The metric must satisfy Einstein equations in every point of spacetime

4\. The geometry will not have event horizons nor singularities

Under these, the general form for the metric tensor is

$ds^{2}=-e^{2\Phi\left(r\right)}c^{2}dt^{2}+\frac{1}{1-\frac{b\left(r\right)}{r}}dr^{2}+r^{2}\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right),$ (1)

where $\Phi\left(r\right)$ is the redshift function and $b\left(r\right)$ is
the form function. When imposing condition 3 above, we obtain the relation
between the functions $\Phi\left(r\right)$ and $b\left(r\right)$ and the
stress-energy tensor that produces the wormhole spacetime geometry. Outside
the wormhole, the spacetime is considered asimptotically flat and have a line
element given by Schwarzschild metric,

$ds^{2}=-\left(1-\frac{2GM}{r^{2}}\right)dt^{2}+\left(1-\frac{2GM}{r^{2}}\right)^{-1}dr^{2}+r^{2}\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right).$ (2)

The junction conditions that follow from the theory of general relativity are
the continuity of the metric components and the extrinsic curvature across the
surface of juncture.

To obtain specific solutions, in wormhole studies is usual to fix a spacetime
geometry (i.e. to fix the functions $\Phi$ and $b$ in order to obtain a
wormhole geometry) and then use the Einstein equations to derive the matter
distribution nedded to obtain the respective metric. Folowing [1, 2] is easy
to see that many wormhole geometries need _exotic matter_ (i.e. matter that
does not obey the null energy condition) to exist, and it is localizated at
the throat of the wormhole. This need is what led Cramer et. al. [5] and
Safanova et. al. [6] to model a wormhole as a point of negative mass to study
gravitational lensing.

###  2.1 Specific Solutions

In this paper we will consider only two specific wormhole solutions obtained
by Lemos et. al. [4]. In both cases we will define $r_{m}$ as the radius of
the throat and $a$ as the radius of the mouth of the wormhole.

####  2.1.1 Solution A

The first solution has an interior metric ($r_{m}\leq r\leq a$) given by

$ds^{2}=-\left(1-\sqrt{\frac{r_{m}}{a}}\right)dt^{2}+\frac{dr^{2}}{\left(1-\sqrt{\frac{r_{m}}{r}}\right)}+r^{2}\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right),$ (3)

and an exterior metric ($a<r<\infty$) given by

$ds^{2}=-\left(1-\frac{\sqrt{r_{m}a}}{r}\right)dt^{2}+\frac{dr^{2}}{\left(1-\frac{\sqrt{r_{m}a}}{r}\right)}+r^{2}\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right).$ (4)

Since the exterior solution must represent Schwarzschild metric, we can
associate a mass to the wormhole,

$M=\frac{\sqrt{r_{m}a}}{2G},$ (5)

and as can be seen , the complete metric is smoothly joined at mouth radius
$a$.

####  2.1.2 Solution B

The second solution has an interior metric ($r_{m}\leq r\leq a$) given by

$ds^{2}=-\left(1-\left(\frac{r_{m}}{a}\right)^{2}\right)dt^{2}+\frac{dr^{2}}{\left(1-\left(\frac{r_{m}}{r}\right)^{2}\right)}+r^{2}\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right),$ (6)

and an exterior metric ($a<r<\infty$) given by

$ds^{2}=-\left(1-\frac{r_{m}^{2}}{ar}\right)dt^{2}+\frac{dr^{2}}{\left(1-\frac{r_{m}^{2}}{ar}\right)}+r^{2}\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right).$ (7)

Again, the metric is smoothly joined at mouth radius $a$, and since the
exterior solution must represent Schwarzschild metric, we associate a mass to
the wormhole,

$M=\frac{r_{m}^{2}}{2aG}.$ (8)

Both metrics represent wormholes connecting two asimptotically flat
spacetimes, and to both of them we associate a positive external mass. This
fact inspire us to think that a light ray that passes near the wormhole but
not into it, will have a deflection similar to that produced by a
Schwarzschild geometry, while a light ray that goes into the wormhole will
have a little different behavior.

##  3 Gravitational Lensing

Because of the arguments given in the preceding section, the gravitational
lensing effects produced by a wormhole will be similar to those produced by a
Schwarzschild metric when the deflected light ray does not enter into the
wormhole, but in order to obtain a mathematical expresion for the deflection
angle of a light ray passing very close to the wormhole we will use the
analytical method for strong field limit gravitational lensing of Bozza [7].

We consider the geometry as follows: A light ray from a source
$\left(S\right)$ is deflected by the wormhole acting as a lens
$\left(L\right)$ and reaches the observer $\left(O\right)$. The background
spacetime is asimptotically flat. The line joining lens and observer
$\left(OL\right)$ is the optic axis. $\beta$ and $\theta$ are the angular
position of the source and the image with respect to the optical axis,
respectively. The distances between observer and lens, lens and source, and
observer and source are $D_{OL}$, $D_{LS}$ and $D_{OS}$, respectively.

The relation between the position of the source and the position of the image
is called the _lens equation_[8],

$\textrm{tan}\theta-\textrm{tan}\beta=\frac{D_{LS}}{D_{OS}}\left[\textrm{tan}\theta+\textrm{tan}\left(\alpha-\theta\right)\right],$
(9)

where $\alpha$ is the deflection angle. For a spherically symmetric spacetime
with line element

$ds^{2}=-A\left(x\right)dt^{2}+B\left(x\right)dx^{2}+C\left(x\right)\left(d\theta^{2}+\textrm{sin}^{2}\theta
d\phi^{2}\right),$ (10)

the deflection angle is given as a function of closest approach
$x_{o}=\frac{r_{o}}{2M}$ by

$\alpha\left(x_{o}\right)=I\left(x_{o}\right)-\pi,$ (11)

where

$I\left(x_{o}\right)=\int_{x_{o}}^{\infty}\frac{2\sqrt{B\left(x\right)}dx}{\sqrt{C\left(x\right)}\sqrt{\frac{C\left(x\right)A\left(x_{o}\right)}{C\left(x_{o}\right)A\left(x\right)}-1}}.$
(12)

The method of Bozza is to expand the integral in $I\left(x_{o}\right)$ around
the photon sphere to obtain a logarithmic expression for
$\alpha\left(x_{o}\right)$, and then, by using the lens equation, obtain the
position of the produced images.

##  4 Deflection Angle for Wormholes

The way to calculate the deflection angle produced by a wormhole depends on
the closest approach of the light ray:

###  4.1 Closest Approach outside Wormhole’s Mouth

When the closest approach of the light ray is greater than wormhole’s mouth
($r_{o}\geq a$), the deflection angle is produced only by the external metric.
Therefore, the lensing effect is, in this case, exactly the same as the
produced by a Schwarzschild metric with the correspondient mass. This angle is
given [7] by

$\alpha\left(x_{o}\right)=-2\textrm{ln}\left(\frac{2x_{o}}{3}-1\right)-0.8056.$
(13)

###  4.2 Closest Approach inside Wormhole’s Mouth

Since wormhole solutions described above are defined in two regions, when the
closest approach of the light ray is inside wormhole’s mouth
($r_{m}<r_{o}<a$), the deflection angle must be calculated as two
contributions: first a deflection produced by the external Schwarzschild-like
metric, and second the contribution produced by the internal metric.

In both of the specific solutions given, the external contribution to the
deflection angle is given as Schwarzschild deflection above but with a closest
approach equal to the wormhole’s mouth,

$\alpha_{e}=-2\textrm{ln}\left(\frac{2a}{3}-1\right)-0.8056.$ (14)

The contribution of the internal metric has to be calculated for each
solution:

####  4.2.1 Solution A

For the specific solution A described above we have

$I_{A}\left(x_{o}\right)=2\int_{x_{o}}^{a}\frac{1}{\sqrt{x^{2}\left(1-\sqrt{\frac{x_{m}}{x}}\right)}}\frac{dx}{\sqrt{\frac{x^{2}}{x_{o}^{2}}-1}}.$
(15)

Making the change $y=\frac{x}{x_{o}}$ we obtain

$I_{A}\left(x_{o}\right)=2\int_{1}^{\frac{a}{x_{o}}}\frac{1}{\sqrt{y^{2}\left(1-\sqrt{\frac{x_{m}}{x_{o}y}}\right)}}\frac{dy}{\sqrt{y^{2}-1}}.$
(16)

The integrand diverges for $y=0$, $y=\frac{x_{m}}{x_{o}}$ and $y=1$. However,
only the third of this values is in the integration range. Because of this, we
will expand the argument of the square root around this divergence up to the
second order in $y$. This gives

$I_{A}\left(x_{o}\right)=2\int_{0}^{\frac{a}{x_{o}}-1}\frac{dz}{\sqrt{\zeta_{A}z+\eta_{A}z^{2}}},$
(17)

where

$\displaystyle z$ $\displaystyle=$ $\displaystyle y-1$ (18)
$\displaystyle\zeta_{A}$ $\displaystyle=$ $\displaystyle
2\left(1-\sqrt{\frac{x_{m}}{x_{o}}}\right)$ (19) $\displaystyle\eta_{A}$
$\displaystyle=$ $\displaystyle 2\left(5-4\sqrt{\frac{x_{m}}{x_{o}}}\right).$
(20)

When $\zeta_{A}$ is non-zero, the leading order of the divergence is
$z^{1/2}$, which can be integrated to give a finite result. When $\zeta_{A}$
vanishes, the divergence is $z^{-1}$ which makes the integral to diverge. In
this way, the condition $\zeta_{A}=0$ give us the radius of the photon sphere,

$x_{ps}=x_{m}.$ (21)

This means that any photon with a closest approach distance $x_{o}=x_{m}$ will
be capturated and it is interesant to note that the radius of the photon
sphere does correspond to the throat radius. Therefore, any photon with a
closest approach distance less than the throat radius does not emerge in the
same part of the universe, but goes into the wormhole to the other mouth.

Integral in (17) can be exactly made as

$I_{A}\left(x_{o}\right)=-\frac{2}{\sqrt{\eta_{A}}}\textrm{ln}\left[\frac{\zeta_{A}}{\zeta_{A}+2\eta_{A}\left(\frac{a}{x_{o}}-1\right)+2\sqrt{\eta_{A}\left(\frac{a}{x_{o}}-1\right)\left(\zeta_{A}+\eta_{A}\left(\frac{a}{x_{o}}-1\right)\right)}}\right].$
(22)

The total deflection angle for solution A is then

$\alpha_{A}\left(x_{o}\right)=\alpha_{e}+I_{A}\left(x_{o}\right).$ (23)

####  4.2.2 Solution B

For the specific solution B, we have

$I_{B}\left(x_{o}\right)=2\int_{x_{o}}^{a}\frac{1}{\sqrt{x^{2}\left(1-\frac{x_{m}^{2}}{x^{2}}\right)}}\frac{dx}{\sqrt{\frac{x^{2}}{x_{o}^{2}}-1}}.$
(24)

This integral can be rewritten as

$I_{B}\left(x_{o}\right)=2\int_{1}^{\frac{a}{x_{o}}}\frac{dy}{\sqrt{y^{2}\left(1-\frac{x_{m}^{2}}{x_{o}^{2}y^{2}}\right)\left(y^{2}-1\right)}},$
(25)

where $y=\frac{x}{x_{o}}$. Once more, the integrand diverges for $y=0$,
$y=\frac{x_{m}}{x_{o}}$ and $y=1$, but we will consider only the third of this
values. Expanding the argument of the square root around this divergence up to
the second order in $y$ we have

$I_{B}\left(x_{o}\right)=2\int_{0}^{\frac{a}{x_{o}}-1}\frac{dz}{\sqrt{\zeta_{B}z+\eta_{B}z^{2}}},$
(26)

where this time

$\displaystyle z$ $\displaystyle=$ $\displaystyle y-1$ (27)
$\displaystyle\zeta_{B}$ $\displaystyle=$ $\displaystyle
2\left(1-\frac{x_{m}^{2}}{x_{o}^{2}}\right)$ (28) $\displaystyle\eta_{B}$
$\displaystyle=$ $\displaystyle 2\left(5-\frac{x_{m}^{2}}{x_{o}^{2}}\right).$
(29)

The condition $\zeta_{B}=0$ give us the radius of the photon sphere, and again
we have

$x_{ps}=x_{m}$ (30)

Therefore the behavior of a photon near this radius is similar to the
described before. The integral in (26) can be exactly made as

$I_{B}\left(x_{o}\right)=-\frac{2}{\sqrt{\eta_{B}}}\textrm{ln}\left[\frac{\zeta_{B}}{\zeta_{B}+2\eta_{B}\left(\frac{a}{x_{o}}-1\right)+2\sqrt{\eta_{B}\left(\frac{a}{x_{o}}-1\right)\left(\zeta_{B}+\eta_{B}\left(\frac{a}{x_{o}}-1\right)\right)}}\right],$
(31)

and the total deflection angle for solution B is

$\alpha_{B}\left(x_{o}\right)=\alpha_{e}+I_{B}\left(x_{o}\right).$ (32)

##  5 Comparison of deflection angles

As we have shown in the preceeding section, wormholes present an interesting
lensing effect. Photons with a closest approach distance greater than
wormhole’s mouth have a Schwarzschild lensing effect, while photons with a
closest approach distance less than wormhole’s mouth have a Schwarzschild
lensing plus an inner lensing effect. Because of this, is important to compare
the deflection angle of wormholes with the deflection of Schwarzschild,

Figure 1. In the left side we have the deflection angle for the wormhole
solution A (continuous) compared with Schwarzschild’s deflection angle
(dotted). Wormhole’s mouth is at $a=2$ (in Schwarzschild units), and the
throat is at $x_{m}=\frac{1}{2}$. It is easy to see that the blue curve
diverges at $x_{o}=x_{m}$. In the right side we hae the deflection angle for
the wormhole solution B (continuous) compared with Schwarzschild’s deflection
angle (dotted). Wormhole’s mouth is again at $a=2$ (in Schwarzschild units)
and the throat is at $x_{m}=\sqrt{2}$. The blue curve diverges exactly at
$x_{o}=x_{m}$. In both cases, outside wormhole’s mouth the angles coincide.

From these graphics we can see that the deflection angle for light passing
outside the wormhole is the same as the one produced by Schwarzschild, and
some differences appear when light goes inside. One of the most important is
the divergence in the blue curves: they occur exactly at wormhole’s throat,
showing that this surface corresponds to a photon sphere.

##  6 Magnification

Magnification is an important characteristic of Gravitational Lensing because
it is easily observable. As is already known, magnification is given by

$\mu=\frac{\mbox{sin}\beta}{\mbox{sin}\theta}\frac{d\beta}{d\theta}.$ (33)

Then, in order to obtain the magnification produced by a wormhole we will use
the produced deflection angle and, using the lens equation (9), obtain a
relation between the angular positions of the image $\left(\theta\right)$ and
the source $\left(\beta\right)$. since the deflection angle is calculated for
each of the presented solutions, the magnification must be evaluated also for
each wormhole.

###  6.1 Solution A

For the first of the presented solutions the magnification is given by:

$\mu=\left(1-\frac{D_{ds}D_{d}}{D_{s}}\sqrt{1-\sqrt{\frac{x_{m}}{a}}}\frac{\alpha\left(x_{o}\right)}{x_{o}}\right)^{-1}\left(1-\frac{D_{ds}D_{d}}{D_{s}}\sqrt{1-\sqrt{\frac{x_{m}}{a}}}\frac{d\alpha}{dx_{o}}\right)^{-1},$
(34)

where $\alpha\left(x_{o}\right)$ is the deflection obtained before and the
derivative $\frac{d\alpha}{dx_{o}}$ can be evaluated using a math software. If
we plot the magnification as a function of the closest aproach we have the
following curves.

Figure 2. Comparison of the magnification for Schwarzschild’s black hole
(left) and wormhole A (right). At this scale the curves look the same.
Wormhole’s mouth is at $a=2$

As it is easily seen, the two curves look exactly the same. Moreover, we can
see the divergence that corresponds to the Einstein’ ring as is expected
because the mouth of the wormhole is situated inside this ring and the
exterior metric is exactly the same as Schwarzschild’s metric.

However if we made a zoom near the wormhole’s mouth, we can see how in the
wormhole’s case, the magnification shows a discontinuity:

Figure 3. Zoom of the magnification for wormhole A, near its mouth located at
$a=2$

It is important to note that the discontinuity occurs exactly at the mouth of
the wormhole, and it happens because here is were the matter distribution of
the wormhole begins. It is also seen a little peak in the magnification curve
inside the wormhole that is not seen in Schwarzschild geometry.

Moreover, when we change the wormhole’s parameters, a very interesting
behavior appears. In the next set of figures we can see the magnification
curve for wormholes with different mouth sizes.

Figure 4.Magnification curves for wormhole A with different mouth’s sizes. In
the first we have $a=2.4$, in the second $a=2.47$, in the third $a=2.6$ and in
the fourth $a=3$.Note that in each case we ohave a discontinuity at $a$.

It is easily seen, that when we change the mouth of the wormhole, the little
peak inside grows. When the mouth of the wormhole is inside the Einstein’s
ring radius the magnification behaves just as commented before, but when the
mouth of the wormhole is greater that the Einstein ring the peak in
magnification of this ring dissapears and there appears a new peak produced by
the wormhole. Observationally, this fact shows an “Einstein’s ring”, shifted
from the original position and this effect can be, in principle, measured.

###  6.2 Solution B

For wormhole B we obtain a magnification given by the expression

$\mu=\left(1-\frac{D_{ds}D_{d}}{D_{s}}\sqrt{1-\frac{x_{m}^{2}}{a^{2}}}\frac{\alpha\left(x_{o}\right)}{x_{o}}\right)^{-1}\left(1-\frac{D_{ds}D_{d}}{D_{s}}\sqrt{1-\frac{x_{m}^{2}}{a^{2}}}\frac{d\alpha}{dx_{o}}\right)^{-1}.$
(35)

Again, we have here the deflection angle for solution B,
$\alpha\left(x_{o}\right)$ and the derivative $\frac{d\alpha}{dx_{o}}$, that
is evaluated using a math software. If we plot the magnification as a function
of the closest approach $x_{o}$, the curve looks exactly the same as the
Schwarzschild magnification (just as for the wormhole soultion a in Figure
2.). However, when we make a zoom of the region near wormhole’s mouth we get a
little difference:

Figure 5. Zoom of the magnification for wormhole B, near its mouth located at
$a=2$

See how we have again a discontinuity localizated at the wormhole’s mouth and
a little peak inside the wormhole that makes the diference when compared with
Schwarzchild’s geometry.Just as in the specific solution B, the inside peak’s
position depends on the size of the wormhole’s mouth. This fact is shown in
the next set of figures:

Figure 6.Magnification curves for wormhole B with different mouth’s sizes. In
the first we have $a=2.4$, in the second $a=2.47$, in the third $a=2.6$ and in
the fourth $a=3$.Note that in each case we ohave a discontinuity at $a$.

Again it is seen an observational difference in the magnification because
there is a peak that does not correspond exactly to the Einstein’s ring but to
the wormhole with a sufficiently greater mouth.

##  7 Conclusion

Gravitational lensing is one of the most important effects in General
Relativity. In this paper we have applied the strong field lensing method to
Morris-Thorne wormholes and the analysis of the deflection angle and
magnification produced by the two specific wormhole solutions described show
that the behaviour of these objects is very similar to the behaviour of
Schwarzschild’s solution. As can be seen from the graphics, there is a
logarithmic divergence of the deflection angle, and there is a photon sphere
that corresponds to the throat of the wormhole. This behavior contrast with
the behaviour reported by Cramer et. al. [5] and Safanova et. al.[6] since
their model for the wormhole is a negative punctual mass. Altough the two
solutions considered here need some exotic (gravitationally negative) mass
inside, the behavior of light deflection does not differ significatively from
the Schwarzschild case.

On the other side, magnification curves for the wormholes are similar to the
obtained for Schwazschild, but when looking in some detail, there is a
discontinuity in the curve (at the point where the mass distribution begins).
It is also of great importance to note here that the size of the mouth of the
wormhole affects the behavior of the magnification curve, and specially when
the mouth of the wormhole is greater than the size of the Einstein’s ring. In
this last case we see how a ring appears but it is not at the expected
position for the Einstein’s ring, making a possible way to identify wormholes
from other mass distributions as, for example, black holes.

## References

  * [1] M. Morris and K. Thorne. Am. Jou. Phys. 56, 395 (1988)
  * [2] M. Morris, K. Thorne and U. Yurtsever. Phys. Rev. Lett. 61, 1446 (1988)
  * [3] M. Visser. _Lorentzian Wormholes: From Einstein to Hawking._ American Institute of Physics. New York. (1995)
  * [4] J. Lemos, F. Lobo y S. Quinet. _Morris-Thorne Wormholes with a Cosmological Constant._ Phys. Rev. D68 064004 (2003)
  * [5] J.G. Cramer, R.L. Forward, M.S. Morris, M. Visser, G. Benford y G.A. Landis. _Natural Wormholes as Gravitational Lenses._ Phys. Rev. D51, 3117 (1995)
  * [6] M. Safanova, D.F. Torres y G.E. Romero. _Microlensing by Natural Wormholes: Theory and Simulations._ Phys. Rev. D65, 023001 (2002)
  * [7] V. Bozza. _Gravitational Lensing in the Strong Field Limit._ Phys. Rev. D66, 103001 (2002)
  * [8] K.S. Virbhadra and G.F.R. Ellis. Phys. Rev. D62, 084003 (2000)


# Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in
Tunisian Dialect

Salima Mdhaffar1, Haroun Elleuch1,2, Fethi Bougares2, Yannick Estève1

1LIA, Avignon University, France
2Elyadata, Paris, France
<EMAIL_ADDRESS>

Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in
Tunisian Dialect

Salima Mdhaffar1, Haroun Elleuch1,2, Fethi Bougares2, Yannick Estève1 1LIA,
Avignon University, France 2Elyadata, Paris, France salima.mdhaffar@univ-
avignon.fr

Abstract

Speech encoders pretrained through self-supervised learning (SSL) have
demonstrated remarkable performance in various downstream tasks, including
Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR).
For instance, fine-tuning SSL models for such tasks has shown significant
potential, leading to improvements in the SOTA performance across challenging
datasets. In contrast to existing research, this paper contributes by
comparing the effectiveness of SSL approaches in the context of (i) the low-
resource spoken Tunisian Arabic dialect and (ii) its combination with a low-
resource SLU and ASR scenario, where only a few semantic annotations are
available for fine-tuning. We conduct experiments using many SSL speech
encoders on the TARIC-SLU dataset. We use speech encoders that were pre-
trained on either monolingual or multilingual speech data. Some of them have
also been refined without in-domain nor Tunisian data through multimodal
supervised teacher-student paradigm. This study yields numerous significant
findings that we are discussing in this paper.

##  1 Introduction

Self-supervised learning methods aim to train a representational model, also
called upstream model, that benefits a collection of downstream tasks. SSL
models are trained by using information extracted from the input data itself
as the label to target. Various techniques have been introduced in the
literature in order to learn powerful representations from the speech signal,
including those based on autoregressive predictive coding Chung et al. (2019),
contrastive losses Schneider et al. (2019); Baevski et al. (2020) and masked
prediction Liu et al. (2021); Chen et al. (2022); Hsu et al. (2021). Other
works explored the combination of contrastive learning and masked language
modeling Chung et al. (2021). The learned SSL models like wav2vec 2.0 Baevski
et al. (2020), wavLM Chen et al. (2022), data2vec Baevski et al. (2022b),
data2vec 2.0 Baevski et al. (2022a), ArTST Toyin et al. (2023), w2v-BERT Chung
et al. (2021) have been proven effective for a wide range of tasks as they
considerably lighten the amount of annotated speech data normally required for
downstream tasks, while exploiting very large amounts of unlabeled data. Some
models have been extended to a cross-lingual setting through XLS-R-128 Babu et
al. (2021), MMS Pratap et al. (2023) and the recently released w2v-BERT 2.0
model Barrault et al. (2023). Several efforts went further to enrich the
frame-level speech representations with textual semantic information. In 2022,
Khurana et al. (2022) proposed such a model called SAMU-XLSR allowing the
obtention of a better semantic encoding in the speech representations of a
pre-trained SSL model. SAMU-XLSR approach follows a teacher-student supervised
learning framework that uses multilingual text/audio paired data and the
Language-agnostic BERT Sentence Embedding (LaBSE) model Feng et al. (2022).
This teacher-student approach is used to refine an SSL pre-trained model
dedicated to speech processing to make it able to produce sentence-level
embeddings from speech similar to the embbedings given by the LaBSE model from
the speech transcriptions. They use the CommonVoice version 8 corpora applied
to 53 languages to refine the XLS-R-128 model Babu et al. (2021).

More Recently, a multilingual sentence embedding space for 200 text and 37
speech languages called SONAR Duquenne et al. (2023) was introduced by META
AI. The authors adopted a two-step training strategy: They first build a
sentence embedding space for the multilingual text before extending this to
the speech modality via a teacher-student approach. Applied to many downstream
tasks, SSL speech encoders have shown their great potential by improving the
state-of-the-art performances on challenging benchmark datasets. However,
understanding speech encoder capabilities requires a benchmarking effort to
compare and draw insights across the techniques.

Indeed, there has been a considerable amount of work and effort in order to
benchmark such SSL models. SUPERB Yang et al. (2021) is a good example of this
effort. It provides a comprehensive speech SSL benchmark including tasks such
as phoneme detection, ASR, slot filling, intent detection, keyword spotting,
etc. The SUPERB benchmark has been extended to evaluate multilingual speech
systems in the ML-SUPERB benchmark Shi et al. (2023). XTREME-S Conneau et al.
(2022) is another example that focuses only on SSL models trained with
multiple languages. Despite the large number of high-quality benchmarks that
evaluate SSL models on various downstream tasks, a limited number of studies
have probed their effectiveness for downstream tasks in spoken Arabic
dialects.

In this paper, we propose to broaden the SSL benchmarking effort to the
Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) of
spoken Arabic Dialects. The contributions of this work are as follows:

  1. 1.

We benchmark the effectiveness of various state-of-the-art SSL speech encoders
in the very challenging context of a low-resource spoken Arabic dialect
(Tunisian dialect) with a limited training data;

  2. 2.

We compare the performances of mono vs. multilingual/bilingual SSL models, and
the impact of a semantic encoding refinement through a multimodal supervised
teacher-student approach;

  3. 3.

We explore the use of recently released SSL models (w2v-BERT 2.0 and SONAR)
for both ASR and SLU. To our knowledge, this is the first work in the
literature to evaluate w2v-BERT 2.0 and SONAR for an SLU task;

  4. 4.

We release our code and
models111https://github.com/speechbrain/speechbrain/tree/develop/recipes/TARIC
for reproducibility and to encourage future research on ASR and SLU of Arabic
dialects.

##  2 Benchmarking protocol

In this section, we formulate the downstream tasks and describe the Tunisian
dialect dataset together with the speech encoders used in our study.

###  2.1 Downstream tasks

In this work, we address two tasks: Automatic Speech Recognition and Spoken
Language Understanding of the Tunisian Arabic spoken dialect.

  1. (a)

ASR: Automatic Speech Recognition has the ultimate goal of providing the
correct transcription given spoken utterance. It is used to assess the ability
of SSL models to extract content information from audio inputs. The ASR task
will be evaluated in terms of Word Error Rate (WER).

  2. (b)

SLU: Spoken language understanding refers to natural language processing tasks
that aim to extract semantic information from speech Tur and De Mori (2011).
Different tasks can be addressed as SLU tasks, such as named entity
recognition from speech, dialog state tracking, intent recognition, slot
filling, etc…In the context of a conventional dialogue system, information is
typically represented through a semantic frame structure. For each utterance,
constructing the semantic representation primarily involves (i) classifying
the user’s utterance in terms of ‘speech acts’ (SA) Searle (1969) or ‘intents’
and (ii) slot filling Wang et al. (2005). We consider these two SLU tasks
following the available annotations: (1) Speech Act classification classifies
speech utterances into predefined classes to determine the intent of the
speaker; (2) Slot filling is a natural language processing and information
extraction technique that entails the identification and extraction of
specific pieces of information or attributes, referred to as ‘slots’, from
unstructured text or spoken language. These slots are typically associated
with predefined categories or entities. The SLU speech act recognition will be
evaluated in terms of Speech Act Error Rate (SAER), which is a standard
classification error rate. The SLU slot filling task will be evaluated in
terms of Concept Error Rate (COER) and Concept/Value Error Rate (CVER). COER
is computed similarly to WER by taking into account only the semantic labels
in the reference and hypothesis annotations. The CVER computation is
identical, but the occurrences of concept/value pairs are taken into account
instead of the concept alone. CVER implies that if any character within the
word’s support prediction or the concept tag prediction differs from the
reference, the entire prediction for that concept is considered as an error.
For the calculation of these metrics, we have drawn on the detailed
description in Laperrière et al. (2022).

###  2.2 Dataset

The ASR dataset TARIC Masmoudi et al. (2014) has been used for this work along
with its recent SLU enrichment, TARIC-SLU Mdhaffar et al. (2024). The
acquisition of the TARIC dataset was carried out in train stations in Tunisia.
The dataset is made of human-human recordings with their manual transcriptions
and semantic annotations. It is composed of more than 2,000 dialogues from 109
different speakers. The dataset222https://demo-lia.univ-avignon.fr/taric-
dataset/ is split into three parts (train, dev and test) as described in Table
1.

Table 1: TARIC-SLU data set split into Train, Dev and Test | Train | Dev | Test
---|---|---|---
#Utterance | 15752 | 771 | 1249
#Dialog | 1713 | 103 | 173
Duration | 7.5 hrs | 29 min | 53 min

TARIC-SLU was annotated using 62 semantic concept tags such as city name
arrival, departure time, ticket price, etc. and 3 speech acts (directives-
answer, directives-query and politeness). The example below shows the original
sentence (a) together with its English translation (b) and the corresponding
semantic annotation (c).

As for the annotation tags, the first word of each sentence represents the
speech act tag ( Directives-query in this example). ‘ < city-name-departure ’
is an opening tag starting the support word sequence ‘Sfax’ and expressing
that this word sequence is associated with the city-name-departure semantic
concept. The character ‘$>$’ represents the closing tag, and it is used to
close all concept tags. The same schema is used for the other semantic
concepts. For speech act tags, there is no closing tag, since each sentence
has a single speech act.

  1. (a)

utf8 باللاهي أعطيني زوز تكايات من صفاقس لتونس مع الثمنية و نصف

  2. (b)

Please give me two tickets from Sfax to Tunis at eight thirty

  3. (c)

Directives-query please  < command-task give me > < number-of-tickets two > <
object tickets > from  < city-name-departure Sfax > to  < city-name-arrival
Tunis > at  < departure-time eight thirty >

###  2.3 SSL speech encoders

Table 2: Architecture details of benchmarked SSL encoders. * SAMU-XLSR is a modified version of XLS-R-128, ♣ all SONAR speech encoders models are a modified version of w2v-BERT, ⋆ represents the size of the paired dataset used in the refinement step. Speech encoder | #Param | #Lang | Hours | Network architecture
---|---|---|---|---
LeBenchmark-7K Evain et al. (2021) | 317M | 1 | 7K | 7CNN-24Trans-Enc
English lv60 Baevski et al. (2020) | 317M | 1 | 960 | 7CNN-24Trans-Enc
HuBERT Hsu et al. (2021) | 316M | 1 | 60K | 7CNN-24Trans-Enc
wavLM Chen et al. (2022) | 316M | 1 | 94K | 7CNN-24Trans-Enc
data2vec 2.0 Baevski et al. (2022a) | 314M | 1 | 960 | 7CNN-24Trans-Enc
VP-100K Wang et al. (2021) | 317M | 23 | 100K | 7CNN-24Trans-Enc
XLS-R-128 Babu et al. (2021) | 317M | 128 | 436K | 7CNN-24Trans-Enc
MMSPratap et al. (2023) | 317M | 1024 | 23K | 7CNN-24Trans-Enc
MMS-1B Pratap et al. (2023) | 1B | 1024 | 23K | 7CNN-48Trans-Enc
w2v-BERT 2.0 Barrault et al. (2023) | 600M | 143 | 4.5M | 2CNN-24Conformer
SAMU-XLSR Khurana et al. (2022) ∗ | 317M | 128 | 436K+12.7K⋆ | 7CNN-24Trans-Enc
SONAR-ARB Duquenne et al. (2023) ♣ | 600M | 2 | 60K+822⋆ | 2CNN-24Conformer
SONAR-ENG Duquenne et al. (2023) ♣ | 600M | 1 | 60K+N/A⋆ | 2CNN-24Conformer
SONAR-FRA Duquenne et al. (2023) ♣ | 600M | 2 | 60K+2K⋆ | 2CNN-24Conformer
Table 3: Architecture details of Whisper models Model | Network architecture | #Params
---|---|---
Whisper-small Radford et al. (2023) | 2-conv 12 Enc-Dec | 244 M
Whisper-medium Radford et al. (2023) | 2-conv 24-Enc-Dec | 769 M

For our study, we used different types of speech encoders. We considered
monolingual SSL speech encoders (French: wav2vec 2.0 LeBenchmark-7K; English:
wav2vec 2.0 LV60, HuBERT, wavLM, data2vec 2.0 and SONAR-ENG) as well as cross-
lingual SSL speech encoders (wav2vec 2.0 VP-100K, XLS-R-128, MMS, MMS-1B,
w2v-BERT 2.0 and SAMU-XLSR) and two bilingual speech encoders (SONAR-ARB and
SONAR-FRA). As explained in the introduction, SAMU-XLSR is a modified version
of XLS-R-128 and SONAR models are a modified version of w2v-BERT 333Notice
that w2v-BERT and w2v-BERT 2.0 are two different models. Both of them are
trained using the same architecture, but w2v-BERT is an English model and
w2v-BERT 2.0 is a multilingual model. w2v-BERT has not been released to the
research community, so we cannot evaluate its performance.. We put the URLs
for all these models in the Appendix A.

SAMU-XLSR is based on the pre-trained multilingual XLS-R-128. SAMU-XLSR will
process audio and text paired data. The XLS-R-128 model used in this approach
was designed to generate speech representations for short 20 milliseconds
speech frames. To make use of this model, SAMU-XLSR performs pooling and
projection to create a single sentence-level representation. In parallel,
LaBSE Feng et al. (2022) sentence-level textual representations are simply
extracted. Both representations being on the same semantic space, SAMU- XLSR’s
is then being pulled towards LaBSE’s with the help of a cosine similarity loss
function. This means the parameters of all SAMU-XLSR’s components are
optimized to predict the textual representations generated by the frozen LaBSE
model. Figure 1 illustrates the training process of SAMU-XLSR.

Figure 1: SAMU-XLSR Training and specialization

SONAR is a new multilingual and -modal text embedding space trained in
encoder-decoder architecture for 200 languages, which substantially
outperforms existing approaches like LASER3 Heffernan et al. (2022) or LaBSE
in multilingual similarity search. Authors apply a teacher-student approach to
extend this embedding space to the speech modality and currently cover 36
languages. Mining is performed in data from publicly available repositories of
web data (tens of billions of sentences) and speech (4 million hours). In
total, to train the model, they align more than 443,000 hours of speech with
texts and create about 29,000 hours of speech-to-speech alignments. Figure 2
illustrates the training process of SONAR.

Figure 2: Sonar architecture

Overall, we experimented with 14 different speech encoders. Table 2 shows all
the details of SSL speech encoder models.
In addition to the aforementioned SSL models, we also evaluated using the
recently released Whisper models from OpenAI Radford et al. (2023). Unlike
self-supervised speech models, Whisper is a multilingual ASR model trained
using a large amount of labelled audio transcription data (680k hours). Using
Whisper was motivated by the recent achievements reported by Wang et al.
(2023). The detailed properties of Whisper models used in this study are
presented in Table 3.

Table 4: Comparison of speech encoders performance across ASR and SLU tasks. Results are reported in WER for ASR and SLU system transcripts. SLU results are reported using COER and CVER for slot filling detection and in terms of SAER for speech act classification. Bold numbers are the best results in all the table. Underlined numbers represent the best results for each colored block (mono-lingual models, cross-lingual models without teacher-student multi-modal training, cross-lingual models with teacher-student multi-modal training, whisper models). FR-7K refers to LeBenchmark-7K, Whisper-S refers to Whisper-Small, Whisper-M refers to Whisper-Medium Speech encoder | COER (SLU) | WER (SLU) | CVER (SLU) | SAER (SLU) | WER (ASR)
---|---|---|---|---|---
| Dev | Test | Dev | Test | Dev | Test | Dev | Test | Dev | Test
FR-7K | 37.78 | 33.16 | 36.04 | 28.61 | 57.57 | 49.57 | 23.87 | 21.14 | 34.80 | 29.69
English lv60 | 36.77 | 32.69 | 37.39 | 30.49 | 57.7 | 50.04 | 26.07 | 21.94 | 35.04 | 30.2
HuBERT | 39.84 | 33.76 | 39.41 | 31.74 | 61.16 | 52.80 | 25.94 | 21.86 | 34.25 | 28.75
WavLM | 35.85 | 32.25 | 34.22 | 27.23 | 55.95 | 50.92 | 24.9 | 21.14 | 32.28 | 26.7
Data2vec 2.0 | 34.50 | 31.8 | 39.41 | 31.71 | 58.94 | 51.15 | 25.16 | 22.9 | 38.58 | 32.1
VP-100K | 35.77 | 31.56 | 35.46 | 27.56 | 56.37 | 48.90 | 24.64 | 22.9 | 32.71 | 26.68
XLS-R-128 | 35.62 | 31.24 | 34.65 | 26.7 | 56.24 | 48.73 | 24.64 | 20.9 | 33.82 | 28.06
MMS | 36.73 | 31.77 | 43.97 | 37.98 | 62.07 | 56.47 | 24.9 | 24.66 | 35.91 | 29.46
MMS-1B | 43.82 | 36.13 | 44.36 | 38.46 | 66.91 | 58.03 | 28.4 | 23.83 | 41.97 | 34.76
w2v-BERT 2.0 | 32.29 | 29.13 | 26.08 | 20.84 | 49.55 | 46.22 | 25.6 | 20.9 | 25.11 | 21.47
SAMU-XLSR | 32.73 | 30.11 | 31.10 | 23.95 | 51.93 | 48.06 | 24.12 | 22.5 | 28.56 | 24.66
SONAR-ENG | 36.58 | 33.59 | 38.34 | 31.58 | 57.15 | 52.33 | 36.71 | 26.10 | 39.77 | 33.67
SONAR-FRA | 36.88 | 33.8 | 38.29 | 30.38 | 58.83 | 53.43 | 34.46 | 27.79 | 39.16 | 32.71
SONAR-ARB | 35.93 | 31.62 | 35.6 | 28.17 | 55.98 | 49.77 | 32.68 | 23.38 | 35.24 | 29.05
Whisper-S | 39.52 | 34.81 | 39.79 | 32.85 | 64.83 | 56.57 | 32.99 | 29.56 | 38.5 | 32.1
Whisper-M | 39.1 | 33.96 | 33.37 | 29.56 | 59.13 | 54.02 | 25.94 | 21.86 | 32.5 | 29.05

##  3 Experiments and Results

###  3.1 Training details

For comparison purposes, we set the same parameters for all the models using
the different speech encoders.

####  3.1.1 ASR

In addition to the speech encoder model, we incorporate an extra layer with
1024 neurons and LeakyReLU as the activation function, followed by a fully-
connected layer and a final $40$-dimensional softmax layer, each dimension
corresponding to a character. The weights of these two additional layers were
randomly initialized, while the weights of the speech encoder part for SSL
models of the neural architecture were initialized using the pre-trained
weights. The fine-tuning is done with the TARIC training set using a
character-level CTC loss function. We optimize the loss with an Adam optimizer
of learning rate = 0.0001 for both speech encoder, and Adadelta with learning
rate = 1.0 for the linear layer.

####  3.1.2 SLU

We formulate the end-to-end SLU task as a character level prediction where
slots are delimited by tag-specific special characters, as in Yadav et al.
(2020); Ghannay et al. (2018); Mdhaffar et al. (2022). We also added the
speech act token to the reference annotation as the first token of each
sequence of words. This way, the end-to-end model learns to both classify the
utterances in terms of speech act, and recognize slot/value pairs present in
the speech segment. As input, the neural network receives a WAV audio file
(PCM, 16 bits, 16kHz, signed integer), and the output is a transcription
enriched with semantic labels and speech acts tags. After processing through
the softmax layer (which has the size of 10644440 characters that cover the
alphabet of the TARIC dataset, 62 characters for slots, one character for
closing slots and three characters for speech acts), the outputs are generated
by a simple greedy decoder. We optimize the loss with an Adam optimizer of
learning rate = 0.0001 for both speech encoder, and Adadelta with learning
rate = 1.0 for the linear layer.

####  3.1.3 Training details for ASR and SLU

We employed a batch size of 4 samples, distributed across 4 NVIDIA V100 32GB
GPU cards. For MMS-1B, we used 4 NVIDIA A100 80GB GPU cards. We utilized two
optimizers: Adadelta for updating the additional layers weights and Adam for
fine-tuning the self-supervised learning (SSL) model. The initial learning
rate for Adadelta was $1.0$, while for Adam it was set to $0.0001$. Our models
were implemented using the SpeechBrain toolkit Ravanelli et al. (2021). The
speech encoders are obtained through the fairseq Ott et al. (2019) framework
for SAMU-XLSR 555SAMU-XLSR is not yet publicly available; it has been kindly
shared by the authors of Khurana et al. (2022), SONAR and data2vec 2.0 and
through HuggingFace for the remaining of models. More details about our models
and the configuration files will be publicly available as a part of the
Speechbrain toolkit.

###  3.2 Results

In this section, we report, analyze and discuss the performance of our models
across various dimensions. All the results, including both ASR and SLU
evaluations, are presented in Table 4. ASR models are evaluated with WER and
SLU models are evaluated using WER (after removing the semantic tags and the
speech act tag from the system output), COER, SAER and CVER. The first part of
the table shows results by using SSL models, and the second part shows results
by using Whisper models. The first part is divided into three sub-parts
according to the type of SSL models: (i) the first sub-part, colored blue for
monolingual models, (ii) the second sub-part, colored pink for cross-lingual
models, and (iii) the third sub-part, colored yellow for the cross-lingual
model after a teacher-student multi-modal semantic training. The second part
of this table is dedicated to the results obtained when using Whisper small
and medium models.

Overall, except for the SLU SAER score on the dev set, the best ASR and SLU
results are obtained when our models are trained using w2v-BERT 2.0 model.
Below, an analysis of the obtained results according to the type of the used
SSL models.

Monolingual models: When we compare the performances of models trained using
monolingual SSL (blue section of the results table), we observe that WavLM is
by far the best-performing model for the ASR task with 32.28% and 26.7% WER
for dev and test sets respectively. This is confirmed as well when evaluating
WER of the SLU model output (column WER (SLU) in table 4). Regarding the SLU
task, while we did not observe any emerged trend, we observed a better
performance of data2vec 2.0 in terms of COER evaluation.

Cross-lingual models without teacher-student multi-modal semantic training:
Results indicate that w2v-BERT 2.0 yields the best performance in both ASR and
SLU tasks (expect for the SAER).

All cross-lingual models: When it comes to cross-lingual models, with (colored
in pink) or without teacher-student multi-modal semantic training (colored in
yellow), setting apart the w2v-BERT 2.0 model, SAMU-XLSR clearly outperforms
VP-100k and MMS based models.

Mono-lingual vs. cross-lingual models: If we compare performances between
monolingual and cross-lingual SSL models, cross-lingual models achieve better
results for the SLU task. Indeed, data2vec 2.0 shows competitive COER scores
while its WER is higher compared to other models. This leads to the conclusion
that while data2vec 2.0 demonstrates the ability to identify semantic
concepts, it falls short in providing accurate transcriptions.

Cross-lingual models with teacher-student refinement:  As stated earlier, the
SLU performances of VP-100K, XLSR-128, and MMS exhibit a consistent trend,
with XLSR-128 showing the highest performance among them. Combining this
multilingual SSL (XLS-R-128) and teacher-student multimodality training in one
model (SAMU-XLSR) provides better results for both ASR and SLU. Shifting the
focus to SONAR models, as expected, SONAR-ARB demonstrates superior
performance compared to both SONAR-ENG and SONAR-FRA for both ASR and SLU
tasks. SONAR models are based on the English version of w2v-BERT.
Unfortunately, the w2v-BERT model used to initialize the speech encoders of
SONAR is not available to assess its contribution.

Whisper models: Results show that Whisper-medium outperform results obtained
using Whisper-small.

Whisper models vs. all SSL models: If we compare performances between Whisper
models (second part of the table) and all SSL models (first part of the
table), almost SSL models achieve better results for the slot filling SLU
task. To transcribe audio files, WER obtained by the ASR or the SLU show
competitive results. For example, ASR results for the dev set (32.5%) obtained
by Whisper medium outperform almost all SSL models except for w2v-BERT 2.0,
SAMU-XLSR and wavLM. This leads to the conclusion that while Whisper models
demonstrates the ability to transcribe audio files, it falls short in
providing accurate semantic extraction compared to the use of SSL speech
encoders.

##  4 Discussion

We carried out some error analyses to quantify some SLU systems performance
across different conditions. We focus on 6 SSL models: LeBenchmark, XLS-R-128,
data2vec 2.0, SAMU-XLSR, w2v-BERT 2.0 and SONAR-ARB.

###  4.1 Acoustic complexity

First, we used the WER of the transcripts produced by the model giving the
best ASR system (w2v-BERT 2.0) to quantify the general complexity of the
utterance. We defined three groups of segments with low (WER $<=$ 20), medium
(20 $<$ WER $<=$ 50) and high (WER $>$ 50) complexity. Figure 3 shows that, as
expected, the SLU performances are overall better for spoken utterances
belonging to the low complexity group. The x-axis represents different levels
of WER, and the y-axis represents COER. However, when we compare the COER of
the SLU systems by group, we observe that w2v-BERT performs better for
segments with low WER. For segments with medium WER, we observe that all the
systems have a comparable behavior except for LeBenchmark: the system shows a
higher COER. For utterances hard to transcribe, SAMU-XLSR is the best one
followed by w2v-BERT 2.0.

Figure 3: SLU performance (COER) across different general complexity levels in
test utterances.

###  4.2 Semantic complexity

In the semantic complexity analysis, we used the number of semantic tags per
utterance in the reference as a proxy of the semantic complexity. Figure 4
shows the results obtained by the evaluated models. The x-axis represents the
number of semantic tags in test utterances, and the y-axis represents COER.
Across the board, utterances with two to six concepts seem to be the easiest.
While data2vec 2.0 and SONAR-ARB perform worse when there are fewer semantic
concepts to extract. They perform best when there are more than six.

Figure 4: SLU performance (COER) across different numbers of semantic tags in
test utterances.

##  5 Conclusion

Our study investigates the usage of various SSL speech encoders for a Spoken
Language Understanding task in challenging circumstances characterized by a
scarcity of both SLU and ASR training data and the low-resource
characteristics of the targeted Tunisian Arabic dialect. Our findings
emphasize the efficacy of SSL pre-trained speech encoders in such conditions,
with notable success observed when employing the w2v-BERT 2.0 model, with 600
millions parameters trained on 4.5M hours of speech from 143 languages.
Additionally, we highlight the noteworthy performance of data2vec 2.0, pre-
trained on English monolingual data, particularly excelling in handling
semantically complex utterances. These outcomes collectively provide valuable
information for advancing SLU methodologies, especially in resource-limited
linguistic contexts. Last, SAMU-XLSR provides very competitive results thanks
to semantic enrichment made by the teacher student approach and in future work
we plan to train a SAMU-w2vBERT 2.0 model to take benefit of the joint SAMU
and w2v-BERT 2.0 capabilities.

##  6 Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (grant
AD011012108R1) and received funding from the EU H2020 SELMA (grant No 957017),
ANR TRADEF project (ANR-22-ASGC-0003) and ESPERANTO research and innovation
programme under the Marie Skłodowska-Curie (grant No 101007666). We would like
to especially thank our colleagues Sameer Khurana and Antoine Laurent for
sharing their SAMU-XSLR model with us.

## References

  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021.  Xls-r: Self-supervised cross-lingual speech representation learning at scale.  pages arXiv–2111.
  * Baevski et al. (2022a) Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. 2022a.  Efficient self-supervised learning with contextualized target representations for vision, speech and language.  In _International Conference on Machine Learning_.
  * Baevski et al. (2022b) Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022b.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  In _International Conference on Machine Learning_ , pages 1298–1312. PMLR.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_.
  * Barrault et al. (2023) Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. 2023.  Seamless: Multilingual expressive and streaming speech translation.  _arXiv preprint arXiv:2312.05187_.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_.
  * Chung et al. (2019) Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019.  An Unsupervised Autoregressive Model for Speech Representation Learning.  In _Proc. Interspeech 2019_.
  * Chung et al. (2021) Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021.  W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.  In _ASRU 2021_.
  * Conneau et al. (2022) Alexis Conneau, Ankur Bapna, Yu Zhang, Min Ma, Patrick von Platen, Anton Lozhkov, Colin Cherry, Ye Jia, Clara Rivera, Mihir Kale, et al. 2022.  XTREME-S: Evaluating Cross-lingual Speech Representations.  In _Proc. Interspeech 2022_ , pages 3248–3252.
  * Duquenne et al. (2023) Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023.  Sentence-level multimodal and language-agnostic representations.  _arXiv preprint arXiv:2308.11466_.
  * Evain et al. (2021) Solène Evain, Manh Ha Nguyen, Hang Le, Marcely Zanon Boito, Salima Mdhaffar, Sina Alisamir, Ziyi Tong, et al. 2021.  Task agnostic and task specific self-supervised learning from speech with lebenchmark.  In _NeurIPS_.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic bert sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891.
  * Ghannay et al. (2018) S. Ghannay, A. Caubrière, Y. Estève, A. Laurent, and E. Morin. 2018.  End-to-end named entity extraction from speech.  _CoRR_ , abs/1805.12045.
  * Heffernan et al. (2022) Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022.  Bitext mining using distilled sentence representations for low-resource languages.  _arXiv preprint arXiv:2205.12654_.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM TASLP_.
  * Khurana et al. (2022) Sameer Khurana, Antoine Laurent, and James Glass. 2022.  Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation.  _Journal of Selected Topics in Signal Processing_.
  * Laperrière et al. (2022) Gaëlle Laperrière, Valentin Pelloin, Antoine Caubrière, Salima Mdhaffar, Nathalie Camelin, Sahar Ghannay, Bassam Jabaian, and Yannick Estève. 2022.  The spoken language understanding media benchmark dataset in the era of deep learning: data updates, training and evaluation tools.  In _LREC_ , pages 1595–1602.
  * Liu et al. (2021) Andy T Liu, Shang-Wen Li, and Hung-yi Lee. 2021.  Tera: Self-supervised learning of transformer encoder representation for speech.  _IEEE/ACM TASLP_.
  * Masmoudi et al. (2014) Abir Masmoudi, Yannick Estève, Mariem Ellouze Khmekhem, Fethi Bougares, and Lamia Hadrich Belguith. 2014.  Phonetic tool for the tunisian arabic.  In _Spoken Language Technologies for Under-Resourced Languages_. Citeseer.
  * Mdhaffar et al. (2024) Salima Mdhaffar, Fethi Bougars, Renato De mori, Salah Zaiem, Mirco Ravanelli, and Yannick Estève. 2024.  Taric-slu: A tunisian dataset for spoken language understanding.  _LREC_.
  * Mdhaffar et al. (2022) Salima Mdhaffar, Valentin Pelloin, Antoine Caubrière, Gaëlle Laperrière, Sahar Ghannay, Bassam Jabaian, Nathalie Camelin, and Yannick Estève. 2022.  Impact analysis of the use of speech and language models pretrained by self-supersivion for spoken language understanding.  In _LREC 2022_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  _arXiv preprint arXiv:1904.01038_.
  * Pratap et al. (2023) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023.  Scaling speech technology to 1,000+ languages.  _arXiv preprint arXiv:2305.13516_.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR.
  * Ravanelli et al. (2021) Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, et al. 2021.  Speechbrain: A general-purpose speech toolkit.  _arXiv preprint arXiv:2106.04624_.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  wav2vec: Unsupervised pre-training for speech recognition.  _Interspeech 2019_.
  * Searle (1969) John R. Searle. 1969.  _Speech Acts: An Essay in the Philosophy of Language_.  Cambridge University Press.
  * Shi et al. (2023) Jiatong Shi, Dan Berrebbi, William Chen, Ho-Lam Chung, En-Pei Hu, Wei Ping Huang, Xuankai Chang, Shang-Wen Li, Abdelrahman Mohamed, Hung-yi Lee, et al. 2023.  Ml-superb: Multilingual speech universal performance benchmark.  _arXiv_.
  * Toyin et al. (2023) Hawau Olamide Toyin, Amirbek Djanibekov, Ajinkya Kulkarni, and Hanan Aldarmaki. 2023.  Artst: Arabic text and speech transformer.  _arXiv preprint arXiv:2310.16621_.
  * Tur and De Mori (2011) Gokhan Tur and Renato De Mori. 2011.  _Spoken language understanding: Systems for extracting semantic information from speech_.  John Wiley & Sons.
  * Wang et al. (2021) Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021.  Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In _ACL_.
  * Wang et al. (2023) Minghan Wang, Yinglu Li, Jiaxin Guo, Xiaosong Qiao, Zongyao Li, Hengchao Shang, Daimeng Wei, Shimin Tao, Min Zhang, and Hao Yang. 2023.  Whislu: End-to-end spoken language understanding with whisper.  In _Proc. Interspeech_ , volume 2023, pages 770–774.
  * Wang et al. (2005) Ye-Yi Wang, Li Deng, and Alex Acero. 2005.  Spoken language understanding.  _Signal Processing Magazine_.
  * Yadav et al. (2020) Hemant Yadav, Sreyan Ghosh, Yi Yu, and Rajiv Ratn Shah. 2020.  End-to-end named entity recognition from english speech.  _Interspeech_.
  * Yang et al. (2021) Shu Wen Yang, Po Han Chi, Yung Sung Chuang, Cheng I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan Ting Lin, et al. 2021.  Superb: Speech processing universal performance benchmark.  In _22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021_ , pages 3161–3165. International Speech Communication Association.

##  Appendix A Appendix: URLs of all models used in this study

For reproducibility of all results obtained in this paper, we put in the table
5 the url for each model.

Table 5: URLs of all models used in this study Speech encoder | URL
---|---
LeBenchmark-7K | https://huggingface.co/LeBenchmark/wav2vec2-FR-7K-large
English lv60 | https://huggingface.co/facebook/wav2vec2-large-lv60
HuBERT | https://huggingface.co/facebook/hubert-large-ll60k
WavLM | https://huggingface.co/microsoft/wavlm-large
Data2vec 2.0 | https://github.com/facebookresearch/fairseq/tree/main/examples/data2vec
VP-100K | https://huggingface.co/facebook/wav2vec2-large-100k-voxpopuli
XLS-R-128 | https://huggingface.co/facebook/wav2vec2-xls-r-300m
MMS | https://huggingface.co/facebook/mms-300m
MMS-1B | https://huggingface.co/facebook/mms-1b
w2v-BERT 2.0 | https://huggingface.co/facebook/w2v-bert-2.0
SAMU-XLSR | SAMU-XLSR is not yet publicly available; it has been shared by Khurana et al. (2022)
SONAR-ENG | https://github.com/facebookresearch/SONAR
SONAR-FRA | https://github.com/facebookresearch/SONAR
SONAR-ARB | https://github.com/facebookresearch/SONAR
Whisper-small | https://huggingface.co/openai/whisper-small
Whisper-medium | https://huggingface.co/openai/whisper-medium