# Dual-angle interferometric scattering microscopy for optical multiparametric
particle characterization

Erik Olsén  Department of Physics, Chalmers University of Technology,
Gothenburg, Sweden Berenice García Rodríguez  Department of Physics,
University of Gothenburg, Gothenburg, Sweden Fredrik Skärberg  Department of
Physics, University of Gothenburg, Gothenburg, Sweden Petteri Parkkila
Department of Physics, Chalmers University of Technology, Gothenburg, Sweden
Giovanni Volpe  Department of Physics, University of Gothenburg, Gothenburg,
Sweden Fredrik Höök  Department of Physics, Chalmers University of Technology,
Gothenburg, Sweden Daniel Midtvedt  Department of Physics, University of
Gothenburg, Gothenburg, Sweden

###### Abstract

Traditional single-nanoparticle sizing using optical microscopy techniques
assesses size via the diffusion constant, which requires suspended particles
in a medium of known viscosity. However, these assumptions are typically not
fulfilled in complex natural sample environments. Here, we introduce dual-
angle interferometric scattering microscopy (DAISY), enabling optical
quantification of both size and polarizability of individual nanoparticles
without requiring a priori information regarding the surrounding media or
super-resolution imaging. DAISY achieves this by combining the information
contained in concurrently measured forward and backward scattering images
through twilight off-axis holography and interferometric scattering (iSCAT).
Going beyond particle size and polarizability, single-particle morphology can
be deduced from the fact that hydrodynamic radius relates to the outer
particle radius while the scattering-based size estimate depends on the
internal mass distribution of the particles. We demonstrate this by optically
differentiating biomolecular fractal aggregates from spherical particles in
fetal bovine serum at the single particle level.

Keywords: holography, iSCAT, nanoparticles, aggregates, size characterization,
mass distribution

Single-nanoparticle characterization in terms of size, shape, and composition
in complex biological environments is a critical challenge within several
research areas, including drug delivery[1], diagnostics[2] and nanosafety[3].
Optical microscopy is in many cases the tool of choice for studying individual
biological nanoparticles due to its high throughput and biological
compatibility[4]. However, although nanoparticles as small as individual
proteins can be detected using label-free optical scattering microscopy[5, 6,
7, 8], multiparametric characterization of individual nanoparticles in terms
of properties such as size, refractive index and morphology remains a
challenge.

Since nanoparticles are smaller than the spatial resolution of optical
scattering microscopy, it is difficult to estimate their size from direct
observation in a microscopy image. Instead, size is typically estimated
indirectly by tracking their position over time, estimating their diffusivity
from their trajectories, and finally using the Stokes-Einstein relation to
relate single-particle diffusivity to the particle size[9]. However, this
requires that the nanoparticles are freely diffusing in a medium with known
viscosity. This imposes critical limitations when the analysis is carried out
in the natural environment of the particles (in situ). For example, the
viscoelastic properties of biological environments are typically complex and
may exhibit spatial variations[10], which severely limits the applicability of
particle sizing using diffusivity.

Quantitative in situ particle characterization using optical microscopy must
instead directly relate the optical scattering of individual particles to
their physical properties. The scattering amplitude depends on the particle
polarizability, defined as

$\alpha\equiv
3V\frac{n_{\text{p}}^{2}-n_{\text{m}}^{2}}{n_{\text{p}}^{2}+2n_{\text{m}}^{2}},$
(1)

where $n_{\text{m}}$ and $n_{\text{p}}$ are the media and particle refractive
indices, respectively, and $V$ is the particle volume. In the limit of small
refractive index differences between the particle and the media, the
polarizability relates to particle properties as $\alpha\approx\left(2/n_{\rm
m}\right)V\Delta n$, where $\Delta n=\left(n_{\rm p}-n_{\rm m}\right)$. Hence,
the polarizability is to a first approximation proportional to the difference
in refractive index between particle and medium. Combined with that $\Delta n$
scales essentially linearly with molecular concentration[11], polarizability
information can be used to estimate particle mass, as demonstrated for both
single biomolecules and living cells[7, 11]. However, since Eq. (1) depends on
both particle volume and $\Delta n$, the scattering amplitude alone is
insufficient to characterize both quantities at once.

In addition to the scattering amplitude, the angular distribution of light
scattering also contains information about particle size and morphology[12].
This forms the basis of particle characterization using multi-angle light
scattering (MALS)[13] and scattering-based flow cytometry[14]. In the context
of microscopy, images of scattering patterns have been employed for
simultaneous estimation of size and refractive index of particles with
diameters down to about half the wavelength of light[15, 16]. This lower size
limit originates from the difficulty of accurately relating a measured
scattering image to particle size for particles near the diffraction limit.

In this work, we introduce dual-angle interferometric scattering microscopy
(DAISY), which offers simultaneous quantification of both size and
polarizability (and hence also $\Delta n$) of individual particles beyond the
limits set by diffraction (around half the wavelength of light) without
requiring precise information about the surrounding medium. DAISY exploits the
fact that the forward and backward scattered optical signals of individual
particles scale differently with size and refractive index, which enables
characterization of particle size and polarizability, given that the optical
signals are measured simultaneously (Figure 1). The forward scattering image
is measured using twilight off-axis holography (Figure 1A), which quantifies
the complex-valued optical field[17], while the backscattering image is
measured using interferometric scattering (iSCAT) microscopy (Figure 1B),
probing the interference between the backscattered particle signal and a
coherent background signal[4] (Supporting Information, Section 1.3). The
twilight holography and iSCAT images (Figure 1C) are processed using standard
algorithms for off-axis holography[18] and a U-Net trained to generate focused
particle images[19] where the signal is proportional to the scattering
amplitude in the backward direction (Supporting Information, Section 1.5),
respectively. After that, the particle signal in each image is estimated by
taking the spatial integral of the post-processed images using a 2D Gaussian
fit.

Figure 1: DAISY working principle. A, B Optical setup to enable simultaneous
twilight off-axis holography (highlighted in A) and interferometric scattering
(iSCAT) (highlighted in B) measurements. Using two different wavelengths for
holography and iSCAT, the two signals are separated by a dichromatic mirror
and directed to two cameras. The low frequency attenuation filter (LFAF)
reduces the amplitude of the unscattered light of the sample beam while having
a negligible effect on the particle signal (Supporting information, Section
1.7). The reduction of the unscattered light is highlighted in the zoomed-in
inset. The LFAF is slightly tilted to direct the reflected light away from the
optical axis. BS: beam splitter, OBJ: objective, TL: tube lens, QWPs: Quarter-
wave plates, and HWPs: Half-wave plates. C After detection, the images of
particle scattering patterns are postprocessed before the signals are
quantified. The two left columns are the scattering patterns after background
subtraction, and the two right columns are the same particles but further
postprocessed, where the twilight images are the average particle signal along
a trace and the iSCAT image has been processed using a U-Net. See Supporting
Information Section 1.5 for more information about the postprocessing. PS:
polystyrene and SiO2: silica. D The ratio between the amplitudes of
backscattered and forward scattered optical fields as a function of particle
radius for spherical particles in water. The gradient color scale encodes the
dependence of the scattering ratio on the particle refractive index from 1.35
to 1.60. The three circles indicate the size and refractive index of the
particles in C.

On the one hand, the optical field in the forward direction is proportional to
the polarizability as (Supporting information, Section 2.3)[20]

$\alpha=\frac{\lambda_{\text{0}}}{n_{\text{m}}\pi}\iint\Im(E_{\text{p}})dA,$
(2)

where $\lambda_{\text{0}}$ is the illumination wavelength in vacuum and
$\Im(E_{\text{p}})$ is the imaginary part of the scattered field in the
forward direction. On the other hand, the optical signal in the backward
direction is proportional to the product between polarizability and self-
interference effects from different scattering elements within the particle,
where the interference effects can be described by the optical form factor $f$
given by Rayleigh-Debye-Gans (RDG) theory[12]. Therefore, the ratio between
the scattering signal in the backward and forward directions is proportional
to the optical form factor in the limits within which RDG theory accurately
describes the particle signal (i.e., $|n_{p}/n_{m}-1|\ll 1$ and
$|n_{p}/n_{m}-1|kR\ll 1$, where $k=2\pi n_{m}/\lambda_{0}$ and $R$ is the
particle radius).

Using RDG theory, the optical form factor of an isotropic particle is[12]

$f(q;\rho)=\frac{1}{V}\int drr^{2}\rho(r)\frac{\sin(qr)}{qr},$ (3)

where $q=(4\pi/\lambda_{0})n_{\rm m}\sin(\theta/2)$, $\theta$ is the angular
difference between the incoming and scattered light, and $\rho(r)$ is the
spatial distribution of scattering elements within the particle. Noticeably,
the optical form factor is independent of the particle refractive index, which
in MALS is used to relate the optical scattering to particle size[13].
Moreover, the optical form factor only depends on the medium refractive index
through the dependence of the $q$-number on $n_{\rm m}$, which enters as a
product with particle radius. Since the refractive index of biological
environments is close to that of water (for example, the refractive index of
the cytoplasm of a cell is only a few percent higher than that of water[21]),
this suggests that particle size can be inferred from the optical form factor
with a precision within a few percent without detailed information about the
surrounding media (Supporting information, Section 2.4).

The size range for which RDG theory accurately describes the optical
scattering depends on the refractive index difference between the particle and
the surrounding media[12]. In the case of spherical particles, Mie theory can
be employed beyond the limitations of RDG. This is illustrated in Figure 1D by
plotting the ratio between the amplitudes of backscattered and forward
scattered optical fields as a function of particle radius for spherical
particles in water. In the case of homogeneous spheres with a known refractive
index and radius less than 170 nm, the scattering ratio is uniquely related to
particle size. Noticeably, this upper size limit coincides with the lower size
limit for which the radius and polarizability can be determined directly from
an off-axis holography image[15]. When the particle refractive index is
unknown, the estimated spread in particle radius based on the scattering ratio
is approximately 3 nm for particles with a radius smaller than 100 nm, even if
the particle refractive index varies between 1.35 and 1.60 (Figure 1D); and
even for particles with a radius of 150 nm, the corresponding uncertainty does
not exceed 10 nm. Thus, DAISY has the potential of extending the lower size
limit image-based particle sizing, with the precision of the size estimate
being dependent on the available refractive index information about the
particle.

We define the DAISY radius (denoted by $r_{\rm DAISY}$) as the smallest radius
of a homogeneous sphere suspended in water displaying the same
backward–forward scattering ratio and polarizability, where the inclusion of
polarizability improves the accuracy when relating the scattering ratio to
size. To extend the particle sizing beyond the limitations of RDG theory, we
here introduce the generalized form factor $\tilde{f}$ as the theoretical
scattering ratio obtained using Mie calculations, normalized such that
$\tilde{f}(R=0)=1$ to make it similar to the optical form factor. The
scattering ratio measured in DAISY can be related to the generalized form
factor as

$\frac{\text{iSCAT}}{|\text{twilight}|}=C\frac{|E_{p}(\text{backward})|}{|E_{p}(\text{forward})|}\equiv
C\tilde{f}(q_{b};\rho,\alpha),$ (4)

where $C$ is a calibration constant obtained by comparing reference
measurements of known particles and $q_{\rm b}$ is the effective wave number
of the iSCAT measurement (Supporting information, Section 1.8). The
generalized form factor is in turn, related to particle size.

Figure 2: Evaluation of the DAISY radius using the simultaneously obtained
hydrodynamic radius as reference. A The DAISY radius with the simultaneously
obtained hydrodynamic radius for two polystyrene samples, one silica sample,
and one mesoporous silica sample, suspended in water. All measured particles
follow a one-to-one relationship with the hydrodynamic radius, where the
median difference between the two size estimates is less than 5% for all
particle populations. The shaded areas correspond to a contour plot of the
DAISY radius and the hydrodynamic radius, where the median value is the point
in the plots. B Refractive index estimates using the polarizability
information from twilight holography and the DAISY radius in A. The estimated
refractive indices are $1.57\pm 0.01$ for polystyrene, $1.42\pm 0.005$ for
silica, and $1.41\pm 0.005$ for mesoporous silica, where the solid line is the
estimated refractive index value and shaded region corresponds to the
uncertainty in the refractive index estimate.

To validate this hypothesis, spherical particles of different sizes and
refractive indices were measured under flow in a microfluidic channel when
suspended in water (Supporting information, Section 1.4). Specifically, two
polystyrene samples ($R=85\pm 13$ nm and $R=105\pm 23$ nm), one silica sample
($R=150\pm 28$ nm), and one mesoporous silica sample ($R=120\pm 29$ nm) were
measured, where the $\pm$ is the standard deviation of the distribution
measured using darkfield nanoparticle tracking analysis (NTA) (Supporting
information, Figure S6). Since these particles were measured freely diffusing
in water, the assumptions underlying diffusivity-based particle sizing are
fulfilled, enabling simultaneous determination of both the hydrodynamic radius
and the DAISY radius.

The estimated DAISY radius and hydrodynamic radius exhibit a one-to-one
correspondence, with a deviation in median size of less than 5% for all
particles, as shown in Figure 2A. Moreover, the distribution widths of the
DAISY radius are consistently similar to or smaller than those of the
hydrodynamic radius. This suggests that the DAISY radius estimation is more
precise than the hydrodynamic radius when the same track length is used
(Supporting information, Section 1.8). Consequently, DAISY effectively
estimates particle sizes below the diffraction limit in microscopy images
without relying on super-resolution imaging or detailed information about the
experimental point spread function.

By using the simultaneously quantified DAISY radius and polarizability,
refractive indices for the measured polystyrene, silica, and mesoporous
particles were determined to be $1.57\pm 0.01$, $1.42\pm 0.005$, and $1.41\pm
0.005$, respectively, as depicted in Figure 2B. Notably, all of these values
are within a 0.02 refractive index difference from prior estimates [22, 23,
24, 5, 25]. This confirms that DAISY enables accurate image-based nanoparticle
characterization in terms of size and polarizability without reliance on the
Stokes-Einstein relation.

To verify that the DAISY radius is indeed insensitive to the precise
information regarding the surrounding media, we measured one particle sample
(polystyrene spheres, modal radius $105$ nm) in aqueous environments with
varying amounts of water and iodixanol, thereby varying the refractive index
of the environment (Figure 3). Even though DAISY radius and polarizability are
estimated as if the particles are in water, the median DAISY radius remains
close to the nominal value of 105 nm and varies by less than 2 nm when the
surrounding refractive index is changed from 1.335 to 1.37 (corresponding to a
variation in iodixanol concentration between 0% and 24%[26]). This low spread
in size estimation is consistent with the observation made in connection with
Eq. (3) regarding that the error in the estimated DAISY-radius is bounded by
the error in the $q$-number used when relating the estimated generalized form
factor to the DAISY radius (Supporting Information, Section 2.4). Given the
width of the DAISY radius distribution, the spread in median DAISY radius
estimates most likely originates from the statistical uncertainty in
estimating the median radius rather than any systematic media refractive index
dependence. Moreover, the particle polarizability estimation decreases with
increasing media refractive index (Figure 3B), as expected from Eq. (2) since
the polarizability is dependent on the refractive index difference between the
particle and media. These results indicate that the particle size estimation
offered by DAISY remains accurate as long as the relative error in the
$q$-number is small ($\ll 1$), which, if combined with the polarizability
information, enables estimation of the refractive index difference between the
particle and the surrounding media.

Figure 3: Evaluation of DAISY radius and polarizability determination in
different media. Box plots of the DAISY radius and polarizability as a
function of the water-iodixanol concentration for 105 nm radius polystyrene
spheres. The DAISY radius remains the same for all different media, whereas
the polarizability decreases as the refractive index difference to the
surrounding media decreases. The DAISY radius and the effective polarizability
are here estimated assuming that the surrounding refractive index has a
refractive index as water. The dashed line is the theoretical polarizability
for a 105 nm radius polystyrene sphere as a function of the surrounding
refractive index.

Note that the DAISY radius is complementary to the hydrodynamic radius, and
these two size estimates coincide in the case of homogeneous spheres. To a
first approximation, the hydrodynamic radius reflects the physical boundary of
the particle, whereas from Eq. (3) the DAISY radius also reflects the interior
mass distribution of the particle. Thus, the relation between the DAISY and
hydrodynamic radius can provide information about the spatial distribution of
mass within the particle and its morphology.

Figure 4: DAISY radius and hydrodynamic radius to distinguish particle
aggregates from solid spheres. A DAISY radius as a function of hydrodynamic
radius for salt-induced aggregation of 35 nm radius polystyrene particles.
From the comparison with the theoretical lines, the DAISY-hydrodynamic radius
relation agrees with that of fractal aggregates with a fractal dimension of
around 2.0. B-C The DAISY radius and hydrodynamics radius for freeze-thawed
induced aggregates of fetal bovine serum (FBS), both B in full serum and C
after size-exclusion chromatography. In A-C, the curved lines correspond to
the theoretical relation for fractal aggregates, and the straight dashed line
is the expected scaling for a solid sphere. The green dashed lines correspond
to a fractal dimension of 2.7, which separates true aggregate detections from
that of spheres with a hydrodynamic radius larger than 200 nm. The insets are
single-particle fractal dimension histograms, where the dashed line
corresponds to the median fractal dimension value. The number of particle
detections in FBS corresponds to a concentration of around $10^{8}$/ml. D
Polarizability as a function of hydrodynamic radius for the data in C, where
the dashed lines are the expected scaling for aggregates with a fractal
dimension of 1.9 and monomer refractive index of 1.53.

To evaluate the potential of DAISY for estimating particle morphology, we
formed aggregates containing 35 nm radius polystyrene spheres via salt-induced
aggregation. The detected aggregates deviate from the one-to-one relation
between the DAISY radius and the hydrodynamic radius previously observed for
homogeneous spheres (Figure 4A), indicating that the morphology of the
aggregates is different from that of homogeneous spheres. To develop this
observation into a quantitative analysis, we first note that particle
aggregates are generally well described as fractal aggregates[27, 28].
Treating the aggregates as spherical units, their mass density is a decaying
function of the radial distance from the aggregate center, $n(r)\sim r^{D_{\rm
f}/3-1}$, where $D_{\rm f}$ is the fractal dimension and is expected to fall
within the range 1.5-2.3 for particle aggregates[29, 30, 15]. An explicit
relation between the DAISY radius and overall aggregate radius, here
approximated by the hydrodynamic radius, can be derived from theoretical
models of fractal aggregates for which the fractal dimension is the only free
parameter (Supporting Information, Section 2.5)[27, 28]. Since the size range
of an unique relation between the optical scattering ratio and particle size
depends on particle morphology (Supporting Information, Section 2.6), we found
that the $(r_{\rm DAISY},r_{\rm H})$-space can be subdivided into two disjoint
regions for the sizes in Figure 4A. One of the regions encompasses spherical,
non-fractal monomeric units (including homogeneous spheres), whereas the other
region encompasses fractal aggregates having fractal dimensions $D_{\rm
f}<2.7$. At this threshold value of the fractal dimension, the theoretical
scattering ratio curves for aggregates and homogeneous spheres tangent each
other, hindering a reliable separation between fractal and non-fractral
structures with fractal dimensions exceeding this value (Supporting
Information, Figure S7). Furthermore, each point within the fractal aggregate
region is related to a specific value of the fractal dimension, indicating
that the fractal dimension of individual aggregates can be quantified based on
their position in $(r_{\rm DAISY},r_{\rm H})$-space. We found that the salt-
induced aggregates generally fall within the region in $(r_{\rm DAISY},r_{\rm
H})$-space encompassing fractal aggregates, validating the analysis approach
outlined above (Figure 4A). The polystyrene aggregates have a population-wide
median fractal dimension of $D_{\rm f}=2.0$, consistent with expectation for
diffusion-limited cluster aggregation (inset to Figure 4A)[29, 30].

To investigate whether the same analysis approach can provide information
about the morphology of the constituents in more complex solutions, we
performed measurements of freeze-thawed fetal bovine serum (FBS) both non-
treated and depleted of proteins (Supporting Information, Section 1.2). In
addition to individual dissolved biomolecules, FBS contains biological
particles such as extracellular vesicles (EVs), lipoprotein particles and
protein aggregates[31]. Using DAISY, we detected particles with a hydrodynamic
radius between 100 and 400 nm in both ordinary FBS and FBS separated from free
proteins using size-exclusion chromatography (Figure 4B-C), at a concentration
of about $10^{8}$/ml. Notably, DAISY’s detection count was $10^{3}$ times
lower than darkfield measurements (Supporting information, Figure S1),
suggesting that it primarily identifies larger particles or aggregates above
its detection limit. The particle detections with a hydrodynamic radius around
150-200 nm have a fractal dimension close to $D_{\rm f}=2.0$, whereas the
larger particles have a fractal dimension in the vicinity of $D_{\rm f}=1.7$.
In addition to this, a small fraction of detections with a hydrodynamic radius
around 100-150 nm deviate from the fractal aggregate scaling, in particular
for the FBS after size-exclusion chromatography, and instead coincide with the
expected scaling for homogeneous spheres (Figure 4C). These detections have a
polarizability of around $0.55\times 10^{6}$ nm3, which together with a
hydrodynamic radius of 125 nm corresponds to a refractive index of about 1.38
(Figure 4D). This value is similar to the expected values for EVs[24, 32, 33],
which, if filled with biological material, are expected to have an optical
form factor similar to homogeneous spheres. However, identification of EV
surface markers is required for conclusive identification of the presence of
EVs. Nevertheless, the rich single-particle shape information using DAISY
indicates that it enables analysis of sub-populations and heterogeneity within
the sample, which extends the possibilities compared to previous works where
particle shape is estimated on the ensemble level from the signal-size
scaling[5, 34, 15].

To gain additional insights into the nature of the fractal aggregate
population of FBS, we investigated the relation between polarizability and
hydrodynamic radius for FBS after size-exclusion chromatography. The
polarizability of a fractal aggregate is directly proportional to the number
of monomers $N$ in the aggregate as $\alpha=\alpha_{0}N$, where $\alpha_{0}$
is the polarizability of the monomers. Since the hydrodynamic radius also
scales with the number of monomers and the fractal dimension is known from the
relation between DAISY radius and hydrodynamic radius, we can estimate the
monomer polarizability. Assuming that the monomer has a refractive index of
1.5, which is similar to lipid bilayers, proteins, and lipoprotein
particles[33, 35, 36], we find that the monomer has a hydrodynamic radius
around 20-30 nm (Figure 4D). This value is considerably larger than individual
proteins (having a typical radius of less than 10 nm)[37]. The estimated
properties of the aggregate are thus consistent with a larger monomer, with
lipoproteins being a likely candidate[33]. However, viral or EV monomers
cannot be excluded based on this data alone[38, 33]. It should be noted that
these results do not exclude the presence of protein aggregates with smaller
monomer units in FBS; it only demonstrates that the aggregates that were
detected in our setup consist of monomers of this size.

In conclusion, we have introduced a versatile method for multiparametric
particle characterization, namely, dual-angle interferometric microscopy
(abbreviated DAISY). We have demonstrated the capacity of DAISY to
simultaneously quantify size and polarizability of particles directly from
optical scattering patterns without being limited by the diffraction limit.
Moreover, the DAISY radius shows negligible dependence on the refractive index
of the surrounding media and is complementary to the hydrodynamic radius,
allowing the combination of the DAISY and hydrodynamic radius to provide
particle morphology estimates. Thus, DAISY opens up for analysis of particle
morphology for freely suspended particles as well as temporally resolved in
situ monitoring of nanoparticles in biological environments. The ability to
measure in biological environments make DAISY a promising candidate for
single-particle analysis of particles inside cells in terms of size and
polarizability, in particular considering that single-particle analysis for
particles inside cells has been previously demonstrated using both holography
and iSCAT separately[39, 40]. Moreover, by analyzing the full scattering
pattern instead of only estimating the integral of the particle images, both
the size range of DAISY and its ability to differentiate different particle
morphologies, such as analyzing asymmetric particles[41], can likely be
further improved. Given DAISY’s versatility and the presented characterization
opportunities, we anticipate that this type of optical microscopy-based
multiparametric characterization will find widespread application in many
areas where nanoparticles play an important role, ranging from industrial
processes to drug discovery and medical diagnostics.

##  1 Acknowledgments

This research was funded by the Swedish research council, grant number
2019-05071, the Knut and Alice Wallenberg Foundation grant number 2019-0577,
and Chalmers Area of Advance Nano. Myfab is acknowledged for support and
access to the nanofabrication laboratories at Chalmers. We also thank Dr.
Björn Agnarsson for fabricating the LFAF.

##  2 Author contributions

EO, FH, and DM conceived the method. EO and BGR implemented the method. EO and
BGR evaluated the method and collected the DAISY data using an experimental
setup and software developed by EO, DM, and FH. EO and DM performed the DAISY
data analysis. PP prepared and analyzed serum samples. FS and DM designed and
trained the neural network. GV, FH, and DM supervised the work. EO and DM
drafted the paper. EO, BGR, GV, FH, and DM drafted the illustrations. All
authors revised the paper.

##  3 Competing interests

The authors declare the following competing financial interest(s): E.O. and
D.M. own shares in a private company (Holtra) that holds IP related to
twilight microscopy method.

##  4 Supplementary information

Materials and methods, data analysis, and theoretical descriptions of the
DAISY radius and the optical scattering ratio for fractal aggregates.

## References

  * [1] A. B. Jindal, “The effect of particle shape on cellular interaction and drug delivery applications of micro-and nanoparticles,” International journal of pharmaceutics, vol. 532, no. 1, pp. 450–465, 2017.
  * [2] G. Chen, I. Roy, C. Yang, and P. N. Prasad, “Nanochemistry and nanomedicine for nanoparticle-based diagnostics and therapy,” Chemical reviews, vol. 116, no. 5, pp. 2826–2885, 2016.
  * [3] A. Zielińska, B. Costa, M. V. Ferreira, D. Miguéis, J. M. Louros, A. Durazzo, M. Lucarini, P. Eder, M. V. Chaud, M. Morsink, et al., “Nanotoxicology and nanosafety: Safety-by-design and testing at a glance,” International Journal of Environmental Research and Public Health, vol. 17, no. 13, p. 4657, 2020.
  * [4] L. Priest, J. S. Peters, and P. Kukura, “Scattering-based light microscopy: From metal nanoparticles to single proteins,” Chemical Reviews, vol. 121, no. 19, pp. 11937–11970, 2021.
  * [5] A. D. Kashkanova, M. Blessing, A. Gemeinhardt, D. Soulat, and V. Sandoghdar, “Precision size and refractive index analysis of weakly scattering nanoparticles in polydispersions,” Nature methods, vol. 19, no. 5, pp. 586–593, 2022.
  * [6] B. Špačková, H. Klein Moberg, J. Fritzsche, J. Tenghamn, G. Sjösten, H. Šípová-Jungová, D. Albinsson, Q. Lubart, D. van Leeuwen, F. Westerlund, et al., “Label-free nanofluidic scattering microscopy of size and mass of single diffusing molecules and nanoparticles,” Nature Methods, pp. 1–8, 2022.
  * [7] G. Young, N. Hundt, D. Cole, A. Fineberg, J. Andrecka, A. Tyler, A. Olerinyova, A. Ansari, E. G. Marklund, M. P. Collier, et al., “Quantitative mass imaging of single biological macromolecules,” Science, vol. 360, no. 6387, pp. 423–427, 2018.
  * [8] M. Dahmardeh, H. Mirzaalian Dastjerdi, H. Mazal, H. Köstler, and V. Sandoghdar, “Self-supervised machine learning pushes the sensitivity limit in label-free detection of single proteins below 10 kda,” Nature Methods, pp. 1–6, 2023.
  * [9] X. Bian, C. Kim, and G. E. Karniadakis, “111 years of brownian motion,” Soft Matter, vol. 12, no. 30, pp. 6331–6346, 2016.
  * [10] Y. M. Efremov, T. Okajima, and A. Raman, “Measuring viscoelasticity of soft biological samples using atomic force microscopy,” Soft matter, vol. 16, no. 1, pp. 64–81, 2020.
  * [11] T. A. Zangle and M. A. Teitell, “Live-cell mass profiling: an emerging approach in quantitative biophysics,” Nature methods, vol. 11, no. 12, pp. 1221–1228, 2014.
  * [12] C. F. Bohren and D. R. Huffman, Absorption and scattering of light by small particles.  John Wiley & Sons, 2008.
  * [13] P. J. Wyatt, “Submicrometer particle sizing by multiangle light scattering following fractionation,” Journal of colloid and interface science, vol. 197, no. 1, pp. 9–20, 1998.
  * [14] E. van der Pol, L. de Rond, F. A. Coumans, E. L. Gool, A. N. Böing, A. Sturk, R. Nieuwland, and T. G. van Leeuwen, “Absolute sizing and label-free identification of extracellular vesicles by flow cytometry,” Nanomedicine: Nanotechnology, Biology and Medicine, vol. 14, no. 3, pp. 801–810, 2018.
  * [15] B. Midtvedt, E. Olsén, F. Eklund, F. Höök, C. B. Adiels, G. Volpe, and D. Midtvedt, “Fast and accurate nanoparticle characterization using deep-learning-enhanced off-axis holography,” ACS Nano, vol. 15, no. 2, pp. 2240–2250, 2021.
  * [16] L. E. Altman and D. G. Grier, “Machine learning enables precise holographic characterization of colloidal materials in real time,” Soft Matter, 2023\.
  * [17] E. Olsén, B. Midtvedt, A. González, F. Eklund, K. Ranoszek-Soliwoda, J. Grobelny, G. Volpe, M. Krzyzowska, F. Höök, and D. Midtvedt, “Label-free optical quantification of material composition of suspended virus-gold nanoparticle complexes,” arXiv preprint arXiv:2304.07636, 2023\.
  * [18] M. K. Kim, “Principles and techniques of digital holographic microscopy,” SPIE Reviews, vol. 1, no. 1, p. 018005, 2010.
  * [19] B. Midtvedt, S. Helgadottir, A. Argun, J. Pineda, D. Midtvedt, and G. Volpe, “Quantitative digital microscopy with deep learning,” Applied Physics Reviews, vol. 8, no. 1, p. 011310, 2021.
  * [20] S. Khadir, D. Andrén, P. C. Chaumet, S. Monneret, N. Bonod, M. Käll, A. Sentenac, and G. Baffou, “Full optical characterization of single nanoparticles using quantitative phase imaging,” Optica, vol. 7, no. 3, pp. 243–248, 2020.
  * [21] B. Gul, S. Ashraf, S. Khan, H. Nisar, and I. Ahmad, “Cell refractive index: Models, insights, applications and future perspectives,” Photodiagnosis and Photodynamic Therapy, vol. 33, p. 102096, 2021.
  * [22] I. D. Nikolov and C. D. Ivanov, “Optical plastic refractive measurements in the visible and the near-infrared regions,” Applied Optics, vol. 39, no. 13, pp. 2067–2070, 2000.
  * [23] X. Ma, J. Q. Lu, R. S. Brock, K. M. Jacobs, P. Yang, and X.-H. Hu, “Determination of complex refractive index of polystyrene microspheres from 370 to 1610 nm,” Physics in medicine & biology, vol. 48, no. 24, p. 4165, 2003.
  * [24] E. van der Pol, F. A. Coumans, A. Sturk, R. Nieuwland, and T. G. van Leeuwen, “Refractive index determination of nanoparticles in suspension using nanoparticle tracking analysis,” Nano letters, vol. 14, no. 11, pp. 6195–6201, 2014.
  * [25] M. A. Odete, F. C. Cheong, A. Winters, J. J. Elliott, L. A. Philips, and D. G. Grier, “The role of the medium in the effective-sphere interpretation of holographic particle characterization data,” Soft Matter, vol. 16, no. 4, pp. 891–898, 2020.
  * [26] T. Boothe, L. Hilbert, M. Heide, L. Berninger, W. B. Huttner, V. Zaburdaev, N. L. Vastenhouw, E. W. Myers, D. N. Drechsel, and J. C. Rink, “A tunable refractive index matching medium for live imaging cells, tissues and model organisms,” Elife, vol. 6, p. e27240, 2017.
  * [27] M. Lattuada, H. Wu, and M. Morbidelli, “Hydrodynamic radius of fractal clusters,” Journal of colloid and interface science, vol. 268, no. 1, pp. 96–105, 2003.
  * [28] C. Sorensen, “Light scattering by fractal aggregates: a review,” Aerosol Science & Technology, vol. 35, no. 2, pp. 648–687, 2001.
  * [29] D. W. Schaefer, J. E. Martin, P. Wiltzius, and D. S. Cannell, “Fractal geometry of colloidal aggregates,” Physical Review Letters, vol. 52, no. 26, pp. 2371–2374, 1984.
  * [30] M. Carpineti, F. Ferri, M. Giglio, E. Paganini, and U. Perini, “Salt-induced fast aggregation of polystyrene latex,” Physical Review A, vol. 42, no. 12, p. 7347, 1990.
  * [31] B. Mannerström, R. O. Paananen, A. G. Abu-Shahba, J. Moilanen, R. Seppänen-Kaijansinkko, and S. Kaur, “Extracellular small non-coding rna contaminants in fetal bovine serum and serum-free media,” Scientific Reports, vol. 9, no. 1, pp. 1–11, 2019.
  * [32] C. Gardiner, M. Shaw, P. Hole, J. Smith, D. Tannetta, C. W. Redman, and I. L. Sargent, “Measurement of refractive index by nanoparticle tracking analysis reveals heterogeneity in extracellular vesicles,” Journal of extracellular vesicles, vol. 3, no. 1, p. 25361, 2014.
  * [33] A. D. Kashkanova, M. Blessing, M. Reischke, A. S. Baur, V. Sandoghdar, and J. Van Deun, “Interferometric nanoparticle tracking analysis enables label-free discrimination of extracellular vesicles from large lipoproteins,” bioRxiv, pp. 2022–11, 2022.
  * [34] D. Midtvedt, F. Eklund, E. Olsén, B. Midtvedt, J. Swenson, and F. Höök, “Size and refractive index determination of subwavelength particles and air bubbles by holographic nanoparticle tracking analysis,” Analytical Chemistry, vol. 92, no. 2, pp. 1908–1915, 2019.
  * [35] P. Parkkila, M. Elderdfi, A. Bunker, and T. Viitala, “Biophysical characterization of supported lipid bilayers using parallel dual-wavelength surface plasmon resonance and quartz crystal microbalance measurements,” Langmuir, vol. 34, no. 27, pp. 8081–8091, 2018.
  * [36] H. Zhao, P. H. Brown, and P. Schuck, “On the distribution of protein refractive index increments,” Biophysical journal, vol. 100, no. 9, pp. 2309–2317, 2011.
  * [37] H. P. Erickson, “Size and shape of protein molecules at the nanometer level determined by sedimentation, gel filtration, and electron microscopy,” Biological procedures online, vol. 11, pp. 32–51, 2009.
  * [38] P. C. Rensen, R. L. de Vrueh, J. Kuiper, M. K. Bijsterbosch, E. A. Biessen, and T. J. van Berkel, “Recombinant lipoproteins: lipoprotein-like lipid particles for drug targeting,” Advanced drug delivery reviews, vol. 47, no. 2-3, pp. 251–276, 2001.
  * [39] B. Midtvedt, J. Pineda, F. Skärberg, E. Olsén, H. Bachimanchi, E. Wesén, E. K. Esbjörner, E. Selander, F. Höök, D. Midtvedt, et al., “Single-shot self-supervised object detection in microscopy,” Nature Communications, vol. 13, no. 1, pp. 1–13, 2022.
  * [40] M. Küppers, D. Albrecht, A. D. Kashkanova, J. Lühr, and V. Sandoghdar, “Confocal interferometric scattering microscopy reveals 3d nanoscopic structure and dynamics in live cells,” Nature Communications, vol. 14, no. 1, p. 1962, 2023.
  * [41] R. Abdulali, L. E. Altman, and D. G. Grier, “Multi-angle holographic characterization of individual fractal aggregates,” Optics Express, vol. 30, no. 21, pp. 38587–38595, 2022.
  * [42] D. Lee, M. Lee, H. Kwak, Y. S. Kim, J. Shim, J. H. Jung, W.-s. Park, J.-H. Park, S. Lee, and Y. Park, “High-fidelity optical diffraction tomography of live organisms using iodixanol refractive index matching,” Biomedical Optics Express, vol. 13, no. 12, pp. 6404–6415, 2022.
  * [43] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” CoRR, vol. abs/1505.04597, 2015.
  * [44] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [45] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015.  Software available from tensorflow.org.
  * [46] R. Parthasarathy, “Rapid, accurate particle tracking by calculation of radial symmetry centers,” Nature methods, vol. 9, no. 7, pp. 724–726, 2012.
  * [47] C. L. Vestergaard, P. C. Blainey, and H. Flyvbjerg, “Optimal estimation of diffusion coefficients from single-particle trajectories,” Physical Review E, vol. 89, no. 2, p. 022726, 2014.


# Hallucinations in Neural Automatic Speech Recognition: Identifying Errors
and Hallucinatory Models

Rita Frieske
HKUST
<EMAIL_ADDRESS>
Bertram E. Shi
HKUST
<EMAIL_ADDRESS>

###### Abstract

Hallucinations are a type of output error produced by deep neural networks.
While this has been studied in natural language processing, they have not been
researched previously in automatic speech recognition. Here, we define
hallucinations in ASR as transcriptions generated by a model that are
semantically unrelated to the source utterance, yet still fluent and coherent.
The similarity of hallucinations to probable natural language outputs of the
model creates a danger of deception and impacts the credibility of the system.
We show that commonly used metrics, such as word error rates, cannot
differentiate between hallucinatory and non-hallucinatory models. To address
this, we propose a perturbation-based method for assessing the susceptibility
of an automatic speech recognition (ASR) model to hallucination at test time,
which does not require access to the training dataset. We demonstrate that
this method helps to distinguish between hallucinatory and non-hallucinatory
models that have similar baseline word error rates. We further explore the
relationship between the types of ASR errors and the types of dataset noise to
determine what types of noise are most likely to create hallucinatory outputs.
We devise a framework for identifying hallucinations by analysing their
semantic connection with the ground truth and their fluency. Finally, we
discover how to induce hallucinations with a random noise injection to the
utterance.

##  1 Introduction

Recent development of neural architectures brought the improvement of
performance among many natural language processing (NLP) tasks, including in
automatic speech recognition (ASR). Deep learning architectures that require
large amounts of data to achieve optimal performance come, however, with
challenges specific to their kind that are not observed in parametric models
Koehn and Knowles (2017). One such challenge is the appearance of
hallucinations, that we define as fluent and coherent outputs of neural models
entirely disconnected from the input.

The phenomenon of hallucinations can be observed among different NLP tasks
such as machine translation, dialogue systems, question answering,
summarization or captioning. Nowadays, with the public deployment of Large
Language Models (LLMs) such as ChatGPT, the topic of hallucinations has gained
vast attention beyond the scientific community (Alkaissi and McFarlane, 2023).
The lack of control over the data that are used to train LLMs increases the
probability of creation of hallucinatory outputs. Furthermore, the blurry
definition of hallucinations and lack of quantitative theoretical frameworks
that would help to distinguish them from other errors impacts the research on
finding methods of their removal. Hallucinations are a dangerous model error
because of their lack of connection with the model input, but fluent and
coherent manner that makes them look like a probable model output. This may
confuse and deceive users and creates lack of trust to the model and, in case
of publicly deployed LLMs - to the industry. Majority of the scientific
articles on hallucinations cover various NLP tasks. There is, however, to our
knowledge no literature coverage on this problem for task such as ASR or
spoken language understanding (SLU). We try to cover this research gap by
addressing specific ways that hallucinations in ASR can be induced through
addition of random noise to the utterance, and explore the ability to detect
hallucinations through the implementation of an algorithm that produces
hallucinations during the testing phase that helps to determine susceptibility
of the model to hallucinations.

Libri 360 | dev-clean | dev-other | test-clean | test-other
---|---|---|---|---
BASELINE WER | 6.14 | 16.44 | 7.27 | 15.65
RR WER | 8.56 | 21.02 | 10.71 | 21.34
RU WER | 6.5 | 17.69 | 6.99 | 17.61
UU WER | 7.25 | 20 | 8.47 | 19.8
UR WER | 14.83 | 39.32 | 16.78 | 38.69

Table 1: Word error rate (WER) of models trained on LibriSpeech 360: baseline
and the models with mismatched labels: repeat-repeat (RR), repeat-unique (RU),
unique-unique (UU) and unique-repeat (UR). The scores show small differences
apart from UR model, which often copies repeated sequences from the training
dataset as a test output.

Errors in ASR are an important measure of the system performance. The main
difference between phonetic ASR errors and hallucinations lies in the severity
of the latter. Phonetic ASR errors appear as badly transcribed words or
phrases, especially when utterances are phonetically similar. They are being
evaluated as a number of phonetic substitutions, insertions and deletions. On
the contrary, the hallucinatory output does not have phonetic or semantic
connection with the source utterance, even though it is often fluent and
coherent. The latter feature makes it especially dangerous in commercially
deployed models, since the end-users may not be monitoring the source
utterance, e.g. during transcription of lectures or speeches (see: table 2 to
compare ASR errors).

| Phonetic Error | Hallucination | Oscillation
---|---|---|---
Reference |  | millimeter roughly
---
one twenty fifth of an inch
| captain lake did not look at all
---
like a london dandy now
indeed ah
Output |  | miller made her roughly
---
one twenty fifths of an inch
| will you let annabel ask her if she
---
sees what it is you hold in your
arms again the voice was soft and wheedling
no annabel said aunt rachel faintly
will you rock again
| a ay indeed ay ay ay ay
---
ay ay ay ay ay ay

Table 2: Examples of differences between the common ASR errors and
hallucinatory content

Similarly to BLEU score, a metric for machine translation (MT) tasks, typical
automatic metrics of an ASR system such as word error rate (WER) do not
reflect the difference between phonetic errors and hallucinations Lee et al.
(2018) (see: table 1). It is therefore important to find the way of capturing
such errors and evaluating models that have increased tendency to generate
them.

We propose a framework to differentiate hallucinations from other ASR errors
based on how semantically related to the reference transcription they are.
Similarly, we assess the fluency of the output sequences to determine whether
the outputs are probable model outputs, or just completely erroneous
transcriptions. Since hallucinations should resemble probable model outputs,
the fluency measure should be high and semantic connection to reference should
be low.

The main contributions of this work are as follows:

  * •

we present the algorithm that assesses whether the model is susceptible to
hallucinations. Our method does not require access to the training dataset

  * •

we implement a framework to identify hallucinations and differentiate them
from ASR phonetic errors, showing a clear distinction between the two error
types

  * •

we present the method to induce hallucinations with random noise injection to
the source utterance

  * •

we show correlation between ASR dataset mismatch and ASR error distributions.
We explore how much label noise is needed to produce given error distributions

  * •

we show that WER is not sensitive towards different ASR error types

Furthermore, to our knowledge, this is the first work on hallucinations
understood as a semantically disconnected and fluent output error of an ASR
system. Understanding hallucinations is crucial to model security in ASR and
NLP tasks.

##  2 Related work

There is no one clear definition of hallucinations in a current body of work,
Ji et al. (2023) distil the definition that spreads across several natural
language generation (NLG) tasks such as dialogue, question answering or
machine translation. They describe hallucinations as "the generated content
that is nonsensical or unfaithful to the provided source content". They
further divide hallucinations into intrinsic and extrinsic. Intrinsic
hallucinations contradict the source content, and extrinsic cannot be verified
from the source content.

The dangerous feature of hallucinations in neural models is their fluency and
coherency, which makes them dangerous in fact-checking, knowledge grounded
dialogue or machine translation Koehn and Knowles (2017).

As per our knowledge, to date, there are no articles on hallucinations in
neural ASR models, given that hallucinations follow the above-mentioned
definition. Serai et al. (2022) use the term "hallucination" in their work in
the context of ASR error creation, rather than referring to the specific type
of error. Similarly, Sagae et al. (2012) use the term "hallucinate" to refer
to producing ASR errors. Concurrently to the literature of hallucinations in
ASR, phonetic ASR errors (in literature referred to just as errors) are well
researched. Anguita et al. (2005) analysed likely confused phone distances in
HMM models, Jyothi and Fosler-Lussier (2010) built a phoneme confusion matrix
to determine how often input text can be confused. Shivakumar et al. (2019)
present multiple example of phonetic errors in ASR as well as the methods of
mitigation, such as lattice rescoring or minimum error rate training (MERT).
Fang et al. (2022) propose method of ASR post-processing error correction by
using non-auto-regressive phonologically based approach.

Evaluation of hallucinatory content is a challenge across different NLP and
ASR tasks. To detect fluent but nonsensical outputs Martindale et al. (2019)
propose BVSS Metric, which is based on cosine similarity and measuring
proportion of information in output that is also in the reference. Errattahi
et al. (2018) lists metrics alternative to WER in ASR such as RIL and its
approximation WIL based on information theory and measure statistical
dependence between the input and output. Regarding fluency measures that could
be helpful to assess hallucinatory output, the proposed methods are mostly
catering to other NLP tasks such as ROUGE metric for summarization Lin (2004).
Although ROUGE is currently the most popular metric, it requires the access to
the parallel corpus. Alternatively, Mutton et al. (2007) propose a metric for
sentence level fluency evaluation called GLEU.

The algorithm implemented in this article is based on the one implemented in
the machine translation context by Lee et al. (2018). Regarding the
experimental noisy models, the original recipe was provided by (Raunak et al.,
2021) for MT. We adjusted the noise amount for the needs of ASR experiment.
The concept of error stimulation with injecting noise to dataset is not new
and in the context of ASR proposed by Fetic et al. (2021) who also implement
similarity metric to evaluate their findings. Morales et al. (2007) add noise
to improve the robustness of the ASR models. Finally, Iwamoto et al. (2022)
show the influence of acoustic artifacts on produced errors by performing
error decomposition analysis.

##  3 Methodology

To explore the hallucinations in ASR, we create models that are prone to
hallucination generation. We do that by creating noisy datasets with
mismatched labels. To evaluate hallucinatory susceptibility of the models, we
compare hallucination distributions before and after random noise injection to
the source utterances of an evaluation set. In order to identify hallucinatory
output, we evaluate its semantic similarity to the reference and its fluency.

###  3.1 Datasets

Dataset Split | Utterances | Hours
---|---|---
LibriSpeech Train 360 | 104014 | 363.6
8% Label Noise | 10000 |  $\sim$34
16% Label Noise | 20000 |  $\sim$68

Table 3: Statistics of the LibriSpeech 360 used to train the models, and of
noisy splits that were added to the main training set. For the experiments
analysing pattern distribution of ASR errors (see: 4.2) the noise subsets were
added to the main training set.

The dataset used for the most of the experiments in this work is a widely used
LibriSpeech dataset Panayotov et al. (2015), that consists of utterances
extracted from the audiobooks. Since processing of speech is computationally
expensive, we use a smaller split of 360 hours of speech out of 960 hour
dataset. For evaluation, we use test and development splits divided into clean
speech, in which the lector’s voice is clear and understandable, as well as
splits marked as ’other’, that are generally more difficult to understand. The
reason behind choosing LibriSpeech is that it is acted speech recorded in
silent environment, therefore it is easy to observe degradation of the result
upon addition of the noise.

Dataset Split | Hours
---|---
CommonVoice 4 | 1,119
CommonVoice DeltaSegment 10.0,12.0,13.0 | 161

Table 4: Statistics of the CommonVoice Dataset

For generalization, we also use a model trained on Common Voice Corpus 4,
English language split, that consists of 1119 hours of validated read speech
from the Wikipedia entries Ardila et al. (2020). For dataset noise through
label mismatch and for evaluation we use Common Voice Delta Segment 10.0,12.0
and 13.0. Since Common Voice dataset is a crowdsource project, the utterance
quality varies from professionally recorded LibriSpeech, which is reflected in
the overall higher WER.

###  3.2 Dataset Noise through Label Mismatch

We explore correlation between ASR error distribution and datasets with
partially noisy mismatched labels. ASR models require huge datasets for
training. Creating a large, balanced and supervised audio dataset is both
expensive, and in cases of low resource languages, very difficult. For
crowdsourced datasets such as CommonVoice, only parts of the dataset are
validated by the users. Lack of audio data can be mitigated with data
augmentation Park et al. (2019), creating synthetic audio data via voice
conversion or text-to-speech (TTS) Thai et al. (2019), and by using one ASR
model to create a training set for another one Ma et al. (2023). Such methods
of increasing data volume might create dataset imbalance that impacts
generation of certain types of errors as a result.

To discover how different types of label mismatch impact error types in ASR,
we follow the dataset mismatch recipe described by Raunak et al. (2021) for
the MT task and create four noisy subsets. The volume of each subset is
approximately 8% or 16% of the original dataset, depending on the experiment.
The type of noise added to the dataset is also a name of each model trained on
it:

  * •

Unique-Unique (UU): random unique source sentences paired with an unrelated
unique random target sentences

  * •

Repeat-Repeat (RR): unique source sentences paired with unrelated unique
random target sentences. 10 such pairs are then repeated, to reach the desired
number of noisy utterances

  * •

Repeat-Unique (RU): repeating source utterances paired with unrelated unique
random target sentences

  * •

Unique-Repeat (UR): unrelated unique random source utterances paired with
repeating targets

The noisy subsets are then added to LibriSpeech 360 dataset. Afterwards, we
use the same transformer model with the same parameters to train a baseline
and four separate noisy models denoted by the above abbreviations.

###  3.3 Evaluation

Figure 1: Word Error Rate (WER) distribution across evaluation dataset of
assessed models. We exclude 0 WER scores since they are not considered error
and for histogram clarity. Although the WER scores alone are not informative
with regard to the types of errors produced by the models, the WER
distribution in UR and RR model shows increase of highly erroneous outputs.

#### ASR Model Evaluation

Hallucinations are not reflected in certain NLP metrics, such as BLEU scores
for machine translation task Lee et al. (2018). To confirm whether
hallucinations have an impact on WER, we compare it across hallucinatory and
non-hallucinatory models. Word error rate is formulated as:

$WER=\frac{S+I+D}{N}\times 100\%$ (1)

Where S indicates a number of substitutions, I stands for insertions, D for
deletions and N is a vocabulary count. Since WER is a distance based ASR
metric, we also compare BLUE and CHRF2 scores as examples of overlapping
n-gram based metrics to validate whether any of them is more sensitive to
certain error types than the other. Finally, we compare a distribution of a
sentence scores across evaluation datasets for a more fine-grained picture of
the error distribution.

#### Semantic Relationship

We use cosine similarity to estimate if the reference and the output are
semantically related. This metric is chosen because its ability to find
sequence vectors that are closest to each other in the Euclidean space without
taking into account the length of each sequence. If o is an output vector and
r is a reference vector, then:

$\cos({\bf o},{\bf r})={{\bf o}{\bf r}\over\|{\bf o}\|\|{\bf
r}\|}=\frac{\sum_{i=1}^{n}{{\bf o}_{i}{\bf r}_{i}}}{\sqrt{\sum_{i=1}^{n}{({\bf
o}_{i})^{2}}}\sqrt{\sum_{i=1}^{n}{({\bf r}_{i})^{2}}}}$ (2)

The sensitivity to the semantics of the sentence and agnosticism towards its
length, makes cosine similarity a good metric to distinguish hallucinations
from oscillations and phonetic errors. We define hallucinations as
semantically disconnected outputs. Therefore, they are the errors with the
lowest cosine similarity and simultaneously high WER.

We characterise ASR phonetic errors by incorrect transcription of particular
sounds that are phonetically similar to the reference word, or are
misinterpreted because of wrong clustering of the spoken utterance. Such
errors return moderately high or high cosine similarity scores, along with a
high WER, and the semantically important part of the sentence remains intact.
To improve detection of phonetic errors, we explored syllable tokenization of
output and reference sentence for more fine-grained cosine similarity score
calculation, but resigned from this approach, since the method falsely assigns
high cosine similarity to long sentences regardless of their semantic
connection.

Oscillations are the errors that often contain the large part or even the
whole correctly transcribed utterance with an addition of a repeating n-gram,
that might extend the length of transcription compared to a reference, which
results in a high WER. Oscillations return high cosine similarity, showing
semantic relationship between the output and the transcription.

#### Fluency

Hallucinations are erroneous, semantically disconnected and fluent outputs.
This third aspect is crucial to differentiate hallucinations from random
repetitions and word salad. To evaluate sentence fluency, we use perplexity,
which gives intuition of how viable is the sentence according to LLM. If P(W)
is a probability of a sentence given language model, then we calculate
perplexity as:

$PPL(W)=\Big{(}\frac{1}{P(W)}\Big{)}^{(\frac{1}{n})}$ (3)

Where n is a number of words in a sentence. Compared with n-gram based fluency
metrics, the advantage of this method is taking the whole context of a
sentence into account. The disadvantage of perplexity as a metric is its lack
of an upper bound, hence we experience the number of outlying values when
comparing large number of data. We compare sentence perplexity scores from two
LLMs: GPT2 and Google’s Flan T5 Small. We find that results returned by Flan
T5 Small contain less extreme values of perplexity values than GPT2, hence the
decision to choose the former model for the fluency evaluation Chung et al.
(2022).

We also explore ROUGE score as another metric for fluency evaluation. This
state-of-the-art summarization metric is often mentioned as the best sentence
fluency indicator Lin (2004).

We find fluency metrics less useful in error identification than cosine
similarity, and hence, we use it as an auxiliary metric to exclude sentences
that are not fluent rather than to define errors.

#### Origins of Hallucinations

Finally, to evaluate the correlation of hallucinations given the training
data, we use distance metrics between hallucinated vector and TF-IDF
representation of the training data. This way, we detect the 5 closest
candidates to each hallucinated sequence. This step is a sanity check to
determine whether the hallucinated sentence or its part is simply copied from
the training dataset due to label repetitions, or it was independently
generated by the model.

###  3.4 Induction of Hallucinations

The volume and the duration of added noise play an important role in creation
of an error distribution that reflects the noise added to the training subset
and therefore allows for the assessment of the model. We propose the following
perturbations applied to the speech:

  * •

random noise injection at the beginning of the sentence. We follow findings of
Lee et al. (2018), who determined that input of a token in the beginning of a
written sentence causes the most cases of hallucinations

  * •

random noise injection into the whole utterance. This method was previously
explored in works on phonetic error generation Iwamoto et al. (2022)

We follow the theory of Feldman (2020) explored by Raunak et al. (2021) who
used perturbations to detect models prone to memorisation that were most
likely to produce hallucinations. We hypothesise that the injected noise
should be strong enough, to deteriorate only a number of overly memorised
sentences, but not as strong as to deteriorate all of them. In both noise
injection scenarios, we explore two digital amplitudes of added noise (0.1 and
0.5 on a scale $[-1,1]$). Furthermore, for noise injection at the beginning of
the sentence, we explore different time increments of noise (0.5s and 1s).

###  3.5 Model Evaluation

We devise a method of finding hallucinations in test time to assess the
hallucinatory susceptibility of the model. To achieve that, we use an
algorithm that is an extension of the idea of Lee et al. (2018) used by them
in the MT task. The algorithm uses perturbations understood as alterations to
the source utterances to assess the model’s potential to produce hallucinatory
content. Specifically, we inject random noise at the beginning of each
utterance to provoke errors. The idea to perturb an utterance at the beginning
comes from the research of Lee et al. (2018), but there, it meant randomly
sampled words in a textual input and not a signal alteration.

Algorithm 1 Hallucination Detection Algorithm

0: Evaluation Data X

0: Hallucinations

$x\in X$ $y\in Y$;

$threshold\ {WER}\ :t_{wer}$;

$threshold\ cosine\ similarity\ :t_{cos}$;

$threshold\ perplexity\ :t_{ppl}$;

while Eval $\hat{y}=model(x)$ do

WER score= WER($\hat{y})$;

if WER score$<t_{wer}$ then

$\hat{y^{\prime}}=perturb(x)$;

WER score’= WER($\hat{y^{\prime}})$;

if WER score’$>t_{wer}$ then

$cosine\ similarity=\cos({\bf\hat{y^{\prime}}},{\bf y})$;

$perplexity=perplexity({\bf\hat{y^{\prime}}})$;

if cosine similarity$<t_{cos}$ and perplexity $<t_{ppl}$ then

hallucinations from perturbation ;

end if

end if

else

if WER score$>t_{wer}$ then

$cosine\ similarity=\cos({\bf\hat{y}},{\bf y})$;

$perplexity=perplexity({\bf\hat{y}})$;

if cosine similarity$<t_{cos}$ and perplexity $<t_{ppl}$ then

natural hallucinations ;

end if

end if

end if

end while

The decision on which utterances should be perturbed is based on a scoring of
the transcribed result, where the low scoring utterances are filtered to avoid
perturbing originally erroneous sequences. After applying perturbations, only
the lowest scoring sequences are evaluated for semantic relationship and for
fluency. To evaluate hallucinatory susceptibility of a model, we compare the
number of filtered hallucinations before and after perturbation and normalize
it with the total vocabulary. We discover that hallucinatory models return
lower amounts of hallucinatory outputs before than after perturbation.
Therefore, the hallucinatory model output returns negative results if
perturbation hallucination scores are deducted from non-perturbed
hallucination scores. This hallucination detection framework, as well as noise
injection and error distribution scoring, are all an extension of the original
algorithm.

##  4 Experiments and Results

###  4.1 Model and hyperparameters

The model used for all the experiments is an off-the-shelf transformer s
provided by fairseq Ott et al. (2019). For model architecture details and
hyperparameters, see: Appendix A.1. As a language model for calculating
perplexity, we use FLAN T5 Small. We use unigram ROUGE for calculating ROUGE
scores.

###  4.2 ASR Error Distribution

We trained 4 models with 8% of the training set mismatched according to the
recipe in section 3.2. The WER, BLUE and CHRF2 scores among different models
were similar, or just slightly worse than the baseline apart from the unique-
repeat (UR) model, that has shown a clear drop in performance (see: Appendix
A.3. After error analysis, we discovered that most of the generated errors in
UR model were the repeating sequences from the noisy subset. We discovered
that the RR model is the most likely to generate oscillations (see: Figure 3),
which correlates with findings of Raunak et al. (2021). Finally, we observed
the increase of hallucinations from perturbations vs without perturbations in
UU model in all the noise injection scenarios. The number of hallucinations
per dataset given 8% of injected noise was small (between 1-2% of all
evaluation datasets, apart from UR in which hallucinations accounted for 8% of
the whole dataset). Given the small amount of hallucinations, we increased the
amount of noise in the dataset to 16% which resulted in clear error patterns.

We observe that the increased amount of repetitive labels in the dataset
correlated with oscillations. We discover that large amounts of repeating
target sequences paired with unique sequences cause hallucinations that are
copies from the repeated training labels. We find that the uniquely mismatched
labels both for source and target sequences (UU model) increase hallucinatory
susceptibility of the model. This susceptibility is not reflected in the
model’s overall score, but the probability of hallucination increase is higher
than in other models when encountered with noisy samples (see: Figure 4).

###  4.3 Hallucination Identification

We discover that the cosine similarity is sensitive towards error
distribution. In most of the analysed cases, we observe clear distinction
between hallucinatory outputs and phonetic errors. After analysing error
distribution peaks, we set the threshold between the hallucinatory output and
phonetic errors as 0.2 cosine similarity.

Figure 2: The cosine similarity and perplexity kernel distribution across 5
models, showing clear distinction between phonetic errors and hallucinations.

We use similar heuristic method to define threshold for WER as 30 and
perplexity as 200 to evaluate fluency of a sentence. The perplexity and ROUGE
as fluency metrics do not indicate error distribution but rather help exclude
outputs that are not fluent and nonsensical such as word salad, repeating
syllables etc.

###  4.4 Hallucination Induction

Dataset label mismatch can increase the number of errors across datasets,
creating general hallucination distribution patterns. To increase probability
of creation of hallucinatory output from the model with small or medium amount
of label noise, which is the most possible real life scenario, the test
utterance should be noisy (see: Appendix A.2)

Figure 3: Comparison between cosine similarity distributions before and after
perturbations in baseline and RR model. The higher mean of cosine similarity
metric in RR model after perturbation suggests an increase of the
oscillations, due to the noise from repetitive labels.

We discover that the noise injection at the beginning of the utterance has the
influence on hallucination distribution, while the noise injected throughout
the whole utterance does not. The noise injected equally throughout the
sentence impacts however performance of the whole evaluation dataset, and it
is reflected in the distribution of phonetic errors (see: Appendix A.4). We
also note that the length and volume of the random noise injected at the
beginning of the sentence has slight influence on the hallucination
production, we chose 1s long random noise perturbation of amplitude 0.5 as the
most successful one with LibriSpeech dataset.

#### Evaluation of the models with Hallucination Detection Algorithm

  * •

baseline phonetic error and hallucination error distribution before and after
perturbation are similar

  * •

UU: phonetic error distribution before and after perturbation is similar. The
hallucination distribution increases after the perturbation, indicating the
hallucination susceptibility of this model. After calculating cosine
similarity between the outputs and TF-IDF representation of the training
dataset, we discovered that the model does not copy labels from the training
set but generates hallucinations unrelated to the content of the training
dataset

  * •

UR: most of the errors are hallucinations that are concentrated at 0.0 cosine
similarity both before and after perturbations, meaning that the sentences are
completely semantically disconnected due to repeating labels. Similar phonetic
error distribution before and after perturbation

  * •

RU: phonetic error distribution before and after perturbation are similar

  * •

RR: different error distribution before and after perturbation. Most of the
errors after perturbation have high cosine similarity even though high WER
which indicates repetitions

Finally, after evaluating the perturbation method on the CommonVoice dataset,
we discover that although it produces hallucinations as expected due to
dataset label noise. The perturbation methods tuned for LibriSpeech dataset
are not strong enough to induce hallucinations in this dataset, due to the
fact that the recording conditions in crowdsourced CommonVoice dataset are
very noisy to begin with. The noise levels of the perturbations need to be
tuned to the respective test set.

Figure 4: The error distribution in baseline and UU model before and after
running the hallucination detection algorithm. The lower peak of the
distribution near 0 cosine similarity indicates hallucinatory outputs. The
increase in hallucination ration in Unique-Unique dataset is not reflected in
its WER results.

###  4.5 Analysis of Source of Hallucination

We analysed the training dataset to find whether some of the hallucinations
are the result of overly memorized phrases. To achieve that, we use TF-IDF
statistic for creating representations of transcribed sentences in the dataset
and compare it using similarity metric, in our case cosine similarity. In each
case, we compare the 5 closest candidates. We discover that most of the
hallucinations from UR model are in fact repeated labels. But other models,
including highly hallucinatory unique-unique, return low similarity scores,
moreover after individual assessment they only sometimes copy certain specific
names or terms, that did not appear in the reference sentence but are
appearing across the training set.

##  5 Conclusion

Hallucinations are the errors generated by neural ASR models. There are
considered to be dangerous because of their seemingly fluid and coherent
manner, but lack of connection with the source. We propose a method to
distinguish hallucination from phonetic ASR errors using combination of
metrics where we implement an algorithm for hallucinatory model classification
without the use of training data. Our method allows detecting hallucinatory
models in test time by estimating numbers of hallucination before and after
noise injection to the utterance. Our method identifies the model that is most
susceptible to hallucinations yet has similar WER to baseline. We further
explore error distributions and correlation between hallucination and random
label mismatch, and oscillations and the occurrence of repetitive labels.

##  6 Limitations

This work researches English ASR, and although the experiments took into
account generalisation across two different datasets, more work regarding
generalisation across different languages and speech oriented tasks such as
SLU should be considered in the future (Hupkes et al., 2023). Finally, the
methods could be re-examined in NLP tasks. The methods were used in supervised
learning scenarios but could be possibly extended to explore hallucinatory
capability of the unsupervised models, i.e. altering the algorithm for
hallucination detection in unsupervised models, such as wav2vec2 (Baevski et
al., 2020).

The thresholds used for identifying hallucinations and phonetic errors are
based on distributions of the errors from the datasets, the heuristic approach
that should be automated for better generalisation capability of our methods.
Furthermore, our method uses previously extracted filterbank features for the
experiments to save preprocessing time during experiments, which limits our
work to experimental setting. The online processing is therefore an important
future improvement of our method. Regarding general understanding of
hallucinations, the limitation of this approach is combining different
automatic metrics together. Creation of a unified metric for measuring
hallucination, especially without the access to the training data, is an
important research direction.

In our work we discuss hallucination detection and identification, however we
do not concentrate on hallucination mitigation, which is also an important
future direction in ASR hallucination research.

## References

  * Alkaissi and McFarlane (2023) Hussam Alkaissi and Samy I McFarlane. 2023.  Artificial hallucinations in chatgpt: Implications in scientific writing.  _Cureus_ , 15.
  * Anguita et al. (2005) J. Anguita, J. Hernando, S. Peillon, and A. Bramoulle. 2005.  Detection of confusable words in automatic speech recognition.  _IEEE Signal Processing Letters_ , 12(8):585–588.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020\.  Common voice: A massively-multilingual speech corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4218–4222, Marseille, France. European Language Resources Association.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Errattahi et al. (2018) Rahhal Errattahi, Asmaa El Hannani, and Hassan Ouahmane. 2018.  Automatic speech recognition errors detection and correction: A review.  _Procedia Computer Science_ , 128:32–37.  1st International Conference on Natural Language and Speech Processing.
  * Fang et al. (2022) Zheng Fang, Ruiqing Zhang, Zhongjun He, Hua Wu, and Yanan Cao. 2022.  Non-autoregressive Chinese ASR error correction with phonological training.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5907–5917, Seattle, United States. Association for Computational Linguistics.
  * Feldman (2020) Vitaly Feldman. 2020.  Does learning require memorization? a short tale about a long tail.  In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_ , STOC 2020, page 954–959, New York, NY, USA. Association for Computing Machinery.
  * Fetic et al. (2021) Raluca Alexandra Fetic, Mikkel Jordahn, Lucas Chaves Lima, Rasmus Arpe Fogh Egebæk, Martin Carsten Nielsen, Benjamin Biering, and Lars Kai Hansen. 2021\.  Topic model robustness to automatic speech recognition errors in podcast transcripts.  _ArXiv_ , abs/2109.12306.
  * Hupkes et al. (2023) Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023.  State-of-the-art generalisation research in nlp: A taxonomy and review.
  * Iwamoto et al. (2022) Kazuma Iwamoto, Tsubasa Ochiai, Marc Delcroix, Rintaro Ikeshita, Hiroshi Sato, Shoko Araki, and Shigeru Katagiri. 2022.  How bad are artifacts?: Analyzing the impact of speech enhancement errors on asr.  In _Interspeech_.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Comput. Surv._ , 55(12).
  * Jyothi and Fosler-Lussier (2010) Preethi Jyothi and Eric Fosler-Lussier. 2010.  Discriminative language modeling using simulated asr errors.  In _Interspeech_.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Lee et al. (2018) Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018.  Hallucinations in neural machine translation.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Ma et al. (2023) Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, and Maja Pantic. 2023.  Auto-avsr: Audio-visual speech recognition with automatic labels.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5.
  * Martindale et al. (2019) Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019.  Identifying fluently inadequate output in neural and statistical machine translation.  In _Proceedings of Machine Translation Summit XVII: Research Track_ , pages 233–243, Dublin, Ireland. European Association for Machine Translation.
  * Morales et al. (2007) Nicolás Morales, Liang Gu, and Yuqing Gao. 2007.  Adding noise to improve noise robustness in speech recognition.  In _Eighth Annual Conference of the International Speech Communication Association_.
  * Mutton et al. (2007) Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. 2007.  GLEU: Automatic evaluation of sentence-level fluency.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 344–351, Prague, Czech Republic. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5206–5210.
  * Park et al. (2019) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019.  Specaugment: A simple data augmentation method for automatic speech recognition.  _Interspeech 2019_.
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021.  The curious case of hallucinations in neural machine translation.  pages 1172–1183.
  * Sagae et al. (2012) K. Sagae, M. Lehr, E. Prud’hommeaux, P. Xu, N. Glenn, D. Karakos, S. Khudanpur, B. Roark, M. Saraçlar, I. Shafran, D. Bikel, C. Callison-Burch, Y. Cao, K. Hall, E. Hasler, P. Koehn, A. Lopez, M. Post, and D. Riley. 2012.  Hallucinated n-best lists for discriminative language modeling.  In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5001–5004.
  * Serai et al. (2022) Prashant Serai, Vishal Sunder, and Eric Fosler-Lussier. 2022.  Hallucination of speech recognition errors with sequence to sequence learning.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:890–900.
  * Shivakumar et al. (2019) Prashanth Gurunath Shivakumar, Haoqi Li, Kevin Knight, and Panayiotis Georgiou. 2019\.  Learning from past mistakes: improving automatic speech recognition output via noisy-clean phrase context modeling.  _APSIPA Transactions on Signal and Information Processing_ , 8:e8.
  * Thai et al. (2019) Bao Thai, Robert Jimerson, Dominic Arcoraci, Emily Prud’hommeaux, and Raymond Ptucha. 2019.  Synthetic data augmentation for improving low-resource asr.  In _2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW)_ , pages 1–9.

##  Appendix A  Appendix

###  A.1 Model Hyper-parameters

Fairseq ASR transformer consists of 12 encoder layers, 6 decoder layers, with
convolutional subsampler and multihead attention with 4 attention heads. The
waveform is represented with filterbank features with 80 filters. Models are
trained on 2 GPUs GeForce RTX 2080 Ti with 10.761 GB memory. Model
hyperparameters learning rate 2e-3, inverse square root for learning rate
scheduler, loss function: label smoothed cross entropy with label smoothing
0.1, warm up of 10000 updates and gradient clipping 10.0, maximum tokens 40000
and maximum update value of 300000. We used sentencepiece with unigram
vocabulary as bpe tokenizer. We used specaugment with frequency mask F 27,
freq mask variable N 2, time mask N of 2, time mask T 100, time mask
probability of 1.0 and no time warping. We also used cepstral mean and
variance normalization. We average 10 last epochs for testing.

###  A.2 Ratio of detected hallucinations in LibriSpeech

Ratio of detected hallucinations per model per evaluation dataset. The noisy
subsets generate more hallucinatory content, hence the intuition of
perturbation as a way to increase probability of hallucinatory output.

Figure 5: Ratio of detected hallucinations per model per evaluation dataset.

###  A.3 Comparison between WER, CHRF2 and BLEU scores

Models do not show influence of types of errors on their performance in WER,
CHRF2 and BLEU scores apart from the UR model which produced hallucinatory
output due to copying labels from the training set.

Table 5: Comparison between WER, BLEU and CHRF2 Scores of baseline and 4
models trained on 8% noisy data.

Libri 360 | WER | BLEU | CHRF2
---|---|---|---
BASELINE | 11.375 | 80.6925 | 90.7225
RR | 15.4075 | 75.6875 | 87.9375
RU | 12.1975 | 78.45 | 89.53
UU | 13.88 | 75.93 | 87.94
UR | 27.405 | 68.6925 | 81.715

###  A.4 Impact of Noise Injection Throughout the Sentence on the Model
Performance

Noise injection in the whole sentence degrades the performance of all the
models.

Figure 6: Figure shows decreased cosine similarity of the outputs after
perturbation with noise injected throughout the sentence.