# Performance of the BIS78 RPC detectors: a new concept of electronics and
detector integration for high-rate and fast timing large size RPCs

L. Pizzimento

###### Abstract

The reduction of the average charge per count in the gas along with the
capability to discriminate very small signals from noise can allow an
efficient and long-term Resistive Plate Chamber detector operation in high
radiation background environment. This goal has been reached during the R$\&$D
program of the BIS78 upgrade project of the ATLAS detector at LHC through the
deep integration of a fast (100 ps peaking time) and sensitive (as small as
100 $\mu$V threshold) Front-End electronics (FE) with a very large size
detector structure. This innovative RPC integration concept pivots on a newly
conceived faraday cage, embedding the readout strips and the FE, tightly
wrapped around a 1 mm gas gap RPC with 1.2 mm thick electrodes, as a fully
independent singlet structure. We studied the performance of BIS78 production
triplet chambers, made of 3 independent singlets of 2 $m^{2}$, each providing
a 2D space and time information, showing a minimum threshold achievable of 3
pC of average charge per count produced inside the gas gap. We show that these
chambers grant a record combined performance of better than $95\%$ single gap
efficiency, time resolution of 350 ps and 10 $kHz/cm^{2}$ rate capability.

##  1 BIS78 ATLAS RPC

The BIS78 project is the pilot project [1] for the barrel inner (BI) plane of
the muon spectrometer upgrade scheduled for the Phase-2 ATLAS upgrade. These
chambers are designed to be compatible with the high luminosity LHC (HL-LHC)
conditions and the BI RPCs will inherit most of the design. The BIS78 RPC
chambers will be deployed in the Phase-1 upgrade, providing a new system which
integrates small-diameter Muon Drift Tube (sMDT) + RPC detectors to be
installed in the barrel-endcap transition region at $1.0<|\eta|<1.3$, which is
the region characterized by the highest background rates in the barrel. The
estimated total rate of the Level-1 single-muon trigger with $p_{T}>20$ GeV is
expected to rise during Run-3 up to $57.6$ $kHz$, while the allocable is $25$
$kHz$ for muon triggers out of a total bandwidth of $100$ $kHz$. In order to
address this problem the pseudorapidity region $1.3<|\eta|<2.5$ will be
covered by the New Small Wheels [1] that will solve the problems of fake
triggers mainly coming from secondary charged particles generated inside the
cryostats and will improve the muon selectivity and the discrimination of low
momentum muons. The region at $1.0<|\eta|<1.3$, which corresponds to the
barrel-endcap transition region, will be covered with the BIS78 RPC chambers,
leading to a total trigger coverage of $\approx 83.5\%$ of the transition
region. The BIS78 stations are formed by two triplets of RPCs coupled to a
multilayer sMDT in the same envelope, without making any electrical contact
between the two systems preserving their electrical and mechanical integrity.
The BIS78 RPC system consists of 16 BIS7 and 16 BIS8 triplets. The total area
covered by this system is 94 $m^{2}$. The new RPCs have a gas gap width of 1
mm. This choice allows to reduce the total thickness of the system and leads
to other benefits: the RPC time resolution scales with the gas gap width. Thus
this reduction leads to an improvement of the time resolution up to $\approx
0.4$ $ns$. The resistive electrodes thickness also has been reduced to $1.2$
$mm$. This reduction takes the advantage of increasing the charge fraction
transferred to the pick-up electrodes, resulting in an improvement of the
Signal-to-Noise ratio. Each detector is readout on both sides by two panels of
orthogonal strips, with strip pitches of $24$-$26$ $mm$ depending on the
chamber type, providing $\eta$ and $\phi$ coordinates. The challenge of making
the chambers compatible with operation at higher hit rates is addressed by a
proportional reduction of charge per count, while simultaneously increasing
the sensitivity and signal-to-noise ratio of the Front-End electronics without
degrading the fast timing features of the RPCs. In order to improve the rate
capability a new Front End electronics has been realized. Also a new Faraday
cage design, suitable for low-threshold operation, is being developed,
allowing a better shielding of the more sensitive Front-End electronics. All
these inter-dependent requirements have led to a full re-design of the RPC,
optimizing it in all aspects, from materials, to readout electronics, up to
chamber layout. The new-generation RPCs will increase the rate capability by
an order of magnitude, decrease the total chamber weight and thickness,
operate at near half the working voltage. The main features of the BIS78-type
RPCs compared to the ATLAS Standard ones are reported in Table 1.

Table 1: RPC Main parameters comparison between ATLAS standard and BIS78 upgrade Detector parameters | ATLAS RPC | BIS78 RPC
---|---|---
Gas gap width | 2 mm | 1 mm
Electrode Thickness | 1.8 mm | 1.2 mm
Time Resolution |  $\approx 1$ ns |  $\approx 0.4$ ns
Space Resolution |  $\approx 6$ mm |  $\approx 1$ mm
Gaps per chamber | 2 | 3
Gas Mixture | ATLAS Standard | ATLAS Standard
Readout | 2D Orthogonal | 2D Orthogonal
FE technology | GaAs | Si$\&$Si-Ge
FE Effective Threshold | 2-3 mV | 0.2-0.3 mV
FE Power consumption | 30 mW/ch | 12 mW/ch

The RPC [2]-[3] rate capability is mainly limited by the current that can be
driven by the high resistivity electrodes. It can be improved by modifying the
highly interconnected parameters which define the voltage drop on the
electrodes. These parameters may be derived by applying the Ohm’s law:

$V_{gas}=V_{a}-R\cdot I=V_{a}-\rho\cdot\dfrac{d}{S}\cdot\braket{Q}\cdot
S\cdot\Phi_{particles}=V_{a}-\rho\cdot d\cdot\braket{Q}\cdot\Phi_{particles}$
(1.1)

Where $V_{gas}$ is the voltage applied across the gas, $V_{a}$ is the total
voltage supplied to the detector, $\rho$ is the bulk resistivity of the
electrodes, $d$ is the electrodes thickness, $\braket{Q}$ is the average
charge per count produced inside the gas, $S$ is the detector area and
$\Phi_{particles}$ represents the flux of ionizing particles. This equation
shows several ways to increase the detectable particle flux, however, the
approach chosen by this project is to reduce the average charge per count
$\braket{Q}$, allowing to increase the rate capability while operating the
detector at fixed current. The drawback of this approach is that reducing the
average charge per count implies a reduction of the injected charge inside the
Front End, hence the signal which needs to be detected will be much smaller.
For this reason, a very sensitive Front End electronics is mandatory in order
to detect such small signal. Moreover, a high suppression of the noise induced
inside the detector by the electronics and by external sources is required,
leading to the chamber structure optimization as a Faraday cage.

The developed Front End electronics board, reported in Figure 1, is composed
of eight channels of a new preamplifier coupled with two full-custom ASIC
discriminators with four channels each, and LVDS transmitters integrated
directly inside the board. The overall features of the preamplifier and of the
ASIC discriminator are reported in the Table 2.

Table 2: Overall properties of preamplifier and full-custom ASIC discriminator

Preamplifier Properties
---
Si standard component |
Preamplifier sensitivity | 0.2-0.4 mV/fC
Power Consumption | 3-5 V 0.5–1 mA
Bandwidth | 100 MHz

Discriminator Properties
---
SiGe full custom |
Discrimination Threshold | 0.5 mV
Power Consumption | 2-3 V 4-5 mA
Bandwidth | 100 MHz

Figure 1: Picture of the Front End electronics board of the BIS78 RPCs.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

##  2 BIS78 RPC performance with cosmic rays

The BIS78 RPC performance has been tested with cosmic rays. The detector
tested is composed of 3 independent mono-gap singlet equipped with the new
Front-End electronics. The gas mixture used is $95\%$ TFE, $4.7\%$
I-$C_{4}H_{10}$, $0.3\%$ SF6. A trigger system has been realized by the usage
of two layers of scintillators, and the tested chamber has been put between
them. The data acquisition has been realized by using a CAEN TDC V1190A with
100 ps time resolution. In Figure 2 a scheme of the experimental setup is
shown.

Figure 2: Sketch of the experimental setup used to test the BIS78 RPCs with
cosmic rays.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

The main parameters monitored in order to study the detector performance with
cosmic rays are the efficiency, the cluster size, the channels noise and the
time resolution.

### Efficiency

The efficiency parameter has been studied in order to verify the overall
behaviour of the detector. Two measurements have been performed checking this
parameter: the efficiency value as a function of the applied High Voltage and
the efficiency map at the knee point. In Figure 3 the efficiency of a singlet
of the BIS78 RPC system is shown.

Figure 3: a) Efficiency of the RPC singlet as a function of the applied High
Voltage. Black and Red markers are related to both the readout panels $\eta$
and $\phi$, while the Blue and the Green ones are the logic AND and OR between
them. b) Map of the efficiency all over the singlet surface estimated at the
knee point (5.5kV).
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

The Figure 3a shows the singlet efficiency measurements as a function of HV;
the red dashed line marks the $95\%$ value at 5.6 kV

The Figure 3b represents the efficiency map taken at the knee point of the
efficiency curve (5.5 kV), in order to be more sensitive to any efficiency
variation. This map has been obtained by dividing the RPC in 8 different zones
(positions) and for each zone the efficiency has been measured. The efficiency
variations reported in the map are within the statistical fluctuations, hence
the efficiency is homogeneous through all the detector surface.

### Cluster Size

Figure 4: Map of the cluster size all over the detector surface at 5.6 kV of
applied High Voltage.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

The Cluster Size parameter has been studied mainly to monitor the self-induced
noise and the crosstalk effects, since those effects would increase the
cluster size to anomalous values. The cluster size mapping has been obtained
by dividing the chamber in 8 positions and for each position the size of
clusters has been monitored. This map allows to verify the absence of any hot
spot and to check the homogeneity of this variable. The cluster size map is
reported in Figure 4.

The map shows an homogeneous behaviour of the cluster size all over the
detector surface with a value completely consistent with the charge induction
phenomenon and a readout strip pitch of $2.5$ cm, implying that there are
neither crosstalk nor self-induced noise effects.

### Channels noise

The noise map allows to highlight problems related to disturbance and broken
channels. This measurement has been performed by using a random trigger and by
counting for each channel the spurious events. The channels counters have been
renormalized according to the whole time acquisition window and to the strip
area. The noise map reported in Figure 5 shows the noise rate in units of
$Hz/cm^{2}$ of both the readout layers $\eta$ and $\phi$. The $\eta$ readout
has an almost homogeneous behaviour with an average noise of 0.3 $Hz/cm^{2}$.
The $\phi$ layer has an average noise rate of 0.5 $Hz/cm^{2}$. The $\phi$
layer is more susceptible to the Low Voltage connection point, which is
located near the channel 64, exhibiting an higher noise rate near this region.

However the overall noise rate for both readout layers is within the
requirements being below 1 $Hz/cm^{2}$.

Figure 5: a) Map of the noise rate of the $\eta$ layer at the working point
(5.6 kV). b) Map of the noise rate of the $\phi$ layer at the working point
(5.6 kV).
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

### Time resolution

The time resolution of the BIS78 RPC detectors has been tested by using the
Time-Of-Flight method between 2 singlets out of 3 (all the combinations have
been checked leading to the same result) and by correcting the discriminator
timewalk effect with the Time-Over-Threshold measurement perfomed by the
Front-End electronics itself [5] (see Figure 6). Moreover, for each readout
channel all the systematic effects have been taken into account and corrected
for. In order to correct the timewalk effect, the function which correlates
the amplitude of the signals with the time in which the signal surpasses the
threshold has been found and corrected for event by event. In Figure 7 the
time resolution of the BIS78 RPC is reported with and without the time walk
correction, achieved at the working point (5.6 kV).

Figure 6: a) Sketch of the Time-Of-Flight method. b) Sketch of the
discriminator timewalk effect.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license. Figure 7: a) Distribution of $\Delta T$ without the timewalk
correction at the detector working point of 5.6 kV. b) Distribution of $\Delta
T$ once the timewalk correction is applied (5.6 kV).
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

The raw BIS78 time resolution is $\sim 400$ ps, while applying the timewalk
correction leads to a slight improvement obtaining a record time resolution of
$\sim 330$ ps with a monogap RPC. The improvement achieved with the timewalk
correction is small due to the already very fast Front-End electronics that
makes the timewalk very small [5].

##  3 BIS78 RPC performance in high radiation environment

The performance of the RPC detector equipped with the newly developed Front-
End electronics under high irradiation has been the crucial point of the GIF++
testbeam at CERN [6]. Moreover, this measurement has been fundamental in order
to verify the improvement in the rate capability achieved by the RPC detector
by means of this new FE electronics. However the FE version used in this test
was not the final one and showed some instabilities in terms of crosstalk and
self induced noise, causing multiple channels to fire simulataneously or to
fire multiple times on the same signal or increasing the discriminated signal
width. Those problems have been solved with the final version of the FE.

This test has been carried on by setting two different sets of FE electronics
parameters. One set is the most performing in terms of efficiency and minimum
achievable threshold, lacking, however, in system stability. The other set is
more conservative in terms of minimum achievable threshold granting on the
other hand high system stability.

Figure 8: Sketch of the experimental setup used at the GIF++ at CERN to test
the BIS78 RPC detector under heavy irradiation.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

The goal of this measurement was the study of the BIS78 RPC performance under
high irradiation; on one hand testing the detector standard working condition
when the conservative set is used and on the other hand checking the highest
performance achievable when the performing parameters are set. The results
achieved by both FE settings and by irradiating the detector with the $14$
$TBq$ ${}^{137}Cs$ source with various shielding factors and by means of a
muon beam are reported. The detector has been positioned at $\sim 5$ m from
the source, its area was 1.8x1.1 $m^{2}$ and the whole experimental setup is
reported in Figure 8.

The singlet I-V curve as function of the absorption factor has been monitored
and it is shown in Figure 9.

Figure 9: I-V curves of the BIS78 RPC singlet as a function of the absorption
factor.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

The measured current can be put in relation to the average charge per count
$\braket{Q}$ and the counting rate $C_{photons}$ on the detector as:

$I=\langle Q\rangle\times C_{photons}$ (3.1)

The current $I$ is normalized with respect to the detector active area, hence
the detector current density ($\mu A/cm^{2}$) is considered. Therefore,
measuring the rate of converted photons $C_{photons}$ and the detector current
with the various source shielding, the average charge $\langle Q\rangle$ per
converted photon can be estimated. Since $C_{photons}$ is also the number of
photons measured by the FE-RPC system, the threshold applied by the Front-End
electronics can be expressed in terms of the average charge per count
delivered inside the detector.

This measurement has been perfomed, considering the current density in the
first higher HV point after the knee of the efficiency plateau. The Front-End
thresholds applied on the charge per count are evaluated by means of the
Equation 3.1.

This study has been performed on both $\eta$ and $\phi$ readouts for different
absorption factors [5] and for both performing and conservative threshold
settings.

Figure 10: a) Efficiency curves of the RPC detector when the conservative
threshold is used, varying the source absorption factor. b) Efficiency curves
of the RPC detector when the performing threshold is set and for different
source absorption factor.
Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

By means of the FE conservative setting an average charge per count produced
inside the gas $\braket{Q}$ of $5\pm 1$ pC was achieved.

The efficiency curves for muons with the conservative settings and for
different absorption factors are reported in Figure 10a. The efficiency curves
do not show any deviation from the source-off behaviour. Operating the
detector in these conditions leads to no efficiency losses for rate of
converted photons up to 2 $kHz/cm^{2}$.

Taking into account the FE performing setting, the estimated FE threshold in
terms of average charge per count on both $\eta$ and $\phi$ readouts is $\sim
3\pm 1$ pC. The big uncertainty is due to the difficulty to correctly estimate
the occupancy and the measured rate, due to system instabilities described
before.

The efficiency curves achieved with the perfoming threshold and with different
absorption factors are reported in Figure 10b. It must be underlined how those
efficiency values, especially for very low absorption factors ($<10Abs$),
represent only the lower limit to the real detector efficiency which is
higher, since the high level of occupancy is the main cause for the flat
efficiency losses. However with the performing threshold set a rate capability
up to 9 $kHz/cm^{2}$ has been achieved, confirming how the Front-End
electronics achieved a $\sim 3$ pC charge threshold on the average charge per
count produced within the gas gap.

##  4 Conclusion

The BIS78 project is the pilot project for the ATLAS detector upgrade of the
barrel muon spectrometer foreseen to meet the requirements of the HL-LHC
running conditions. The BIS78 RPCs detectors grant a record combined
performance of better than $95\%$ single gap efficiency and time resolution of
$\sim 330$ $ps$. The highest rate capability achieved by this system is $9$
$kHz/cm^{2}$ with an efficiency above $80\%$. Further tests are ongoing in
order to fully demonstrate that the new version of FE addresses the
instability problems leading to a fully efficient detector with such high rate
capability. Moreover, the succesfulness of the integration of the FE
electronics within a large area detector has been demonstrated, reaching high
performance along with a good homogeneity overall behaviour through all the
BIS78 modules produced.

## References

  * [1] ATLAS Collaboration, _Technical Design Report for the Phase-II Upgrade of the ATLAS Muon Spectrometer, CERN-LHCC-2017-017, ATLAS-TDR-026 (2017)_.
  * [2] R. Santonico and R. Cardarelli, _Nucl. Instr. And Meth._ 187 (1981) 377.
  * [3] R. Santonico, R. Cardarelli, A. Di Biagio, A. Lucci, _Nucl. Instr. And Meth._ A263 (1988) 20.
  * [4] L. Pizzimento, _Development of a new Front End electronics in Silicon and Silicon-Germanium technology for the Resistive Plate Chamber detector for high rate experiments_ , JINST 14, C10010.
  * [5] L.Pizzimento, _Development and integration study of a new Front-End electronics for the upgrade of the Resistive Plate Chamber detector for high radiation environment_ , PhD thesis at University of Rome TorVergata, XXXII ciclo del corso di dottorato
  * [6] CERN Secondary Beam and Areas, _http://cern.ch/sba_


# High-Precision Voice Search Query Correction via Retrievable Speech-Text
Embeddings

###### Abstract

Automatic speech recognition (ASR) systems can suffer from poor recall for
various reasons, such as noisy audio, lack of sufficient training data, etc.
Previous work has shown that recall can be improved by retrieving rewrite
candidates from a large database of likely, contextually-relevant alternatives
to the hypothesis text using nearest-neighbors search over embeddings of the
ASR hypothesis text to correct and candidate corrections. However, ASR-
hypothesis-based retrieval can yield poor precision if the textual hypotheses
are too phonetically dissimilar to the transcript truth. In this paper, we
eliminate the hypothesis-audio mismatch problem by querying the correction
database directly using embeddings derived from the utterance audio; the
embeddings of the utterance audio and candidate corrections are produced by
multimodal speech-text embedding networks trained to place the embedding of
the audio of an utterance and the embedding of its corresponding textual
transcript close together. After locating an appropriate correction candidate
using nearest-neighbor search, we score the candidate with its speech-text
embedding distance before adding the candidate to the original n-best list. We
show a relative word error rate (WER) reduction of 6% on utterances whose
transcripts appear in the candidate set, without increasing WER on general
utterances.

Index Terms: Embeddings, End-to-End ASR, Contextual ASR, Nearest Neighbors,
Retrieval

##  1 Introduction

ASR contextualization (or biasing) systems improve recognition accuracy for
queries when additional information is available, e.g., the dialog state,
device state, or the language of the query. For example, the correct
hypothesis for a query to a voice assistant device in a music-playing state is
more likely to contain musician “Eminem” than the acoustically identical, but
linguistically incorrect, alternative “M&M.”

ASR correction is a modular approach to ASR contextualization which allows for
the use of the full ASR output and other resources which may not be available,
or may be too expensive to access, during first-pass recognition. In
particular, in many domains, it is desirable to use very large collections of
correction candidates — for example, for media queries, we can take advantage
of large databases of artists, songs, and albums. We may quickly find
correction candidates in a large database as follows:

  1. 1.

A “dual encoder” model consisting of two encoder networks (or a single shared
encoder) is trained to map the target for correction (e.g., ASR hypothesis
text) and candidate corrections (e.g., database entries) into the same
embedding space (as in, e.g., [1]), such that incorrect text is close to the
corresponding correction in the embedding space. For the correction task, this
means that incorrect recognition hypotheses and phonetically-similar
corrections are mapped to nearby points in the embedding space.

  2. 2.

At inference, we produce embeddings of the correction target and candidate
corrections, and a fast approximate neighbors system (e.g., ScaNN [2]) is used
to find candidate corrections in the correction candidate database at
inference time. Note that, if the set of candidate corrections is known ahead
of time (e.g., as in the case of a media entity database), the embeddings of
the candidate corrections can be computed before inference.

ASR correction using nearest neighbor embedding search in large candidate
databases was previously demonstrated using the existing top ASR hypothesis’s
text as the embedder input for retrieving similar phrases from a database of
(embedded) textual candidate corrections [3]. However, the textual hypothesis
may be far from the phonetic ground truth; using a phonetically-inaccurate
hypothesis to retrieve nearest neighbors for hypothesis rewriting can lead to
low precision, since the search key does not accurately reflect the utterance
audio. To overcome this limitation, we demonstrate a contextual ASR hypothesis
correction system based on _multimodal_ speech-text embeddings with the
following improvements over text-based approaches:

  * •

Similar phrase retrieval directly from utterance audio
Our nearest-neighbor retrieval is based on query embeddings computed directly
from the utterance audio, eliminating the imprecision that may be introduced
by phonetically-inaccurate ASR hypotheses. Further, the encoders that compute
these embeddings can be trained specifically to the ASR engine, language,
application domain, etc. [3]

  * •

Large-scale, modular, efficient short-form ASR correction
Our approach can be applied on top of a frozen base ASR model. Computing
fixed-size embeddings enables efficient matching against a large collection
(up to 128K entries in this paper) of contextually-relevant phrases using fast
approximate-nearest-neighbors search [2]. For voice search, using a single
collection simplifies contextual ASR correction, since the phrases can be
applied across different contextual situations (e.g., device is playing music,
timer active, etc.). More details about how we select such phrases are in
Section 4.3.

Fig. 1: Overall system. An offline job builds the embeddings database of
retrievable phrases. The utterance audio is used to retrieve nearest
neighbors. During N-best list expansion, the nearest neighbor phrase is scored
and unioned with the original n-best list.

##  2 Related work

Classical contextualization approaches for hybrid ASR use weighted finite
state transducers (WFSTs) to bias contextually-relevant phrases by
interpolating their base LM costs with those from a WFST-encoded external
language model (LM) during decoding [4], with optional search space expansion
to increase the likelihood that potentially-biased phrases survive beam
pruning [5]. For E2E ASR systems without language models, shallow fusion
approaches have been proposed, as in [6]. These approaches are limited by the
ability to surface contextually-relevant phrases during first-pass beam
search, limits on the sizes of WFSTs that can be effectively constructed and
traversed during search, and the lack of language models in pure E2E ASR
systems. In pure E2E ASR, attention-based biasing approaches [7, 8, 9] avoid
the need for a language model for biasing and mitigate search problems by
integrating biasing context directly into the model’s acoustic and/or
linguistic representations; however, such approaches also run into scaling
limits as the number of phrases grows. Similarly, integrating k-NN search with
language models [10] allows adding variable contextually-relevant n-grams at
inference time, potentially improving recall at the cost of tighter
integration with first-pass ASR. Our retrieval approach is independent of
first-pass beam search, so that phrase recall is independent of first-pass
performance, does not require a tight integration between biasing and ASR, and
scales to hundreds of thousands of phrases using simple mechanisms.

Typical ASR correction models operate on the lattice, using WFST operations in
the phonemes space to find correction candidates [11, 12], or on ASR
hypotheses, e.g., using sequence-to-sequence transducers to correct instances
of contextually-relevant phrases [13]; recent work has extended textual
spelling correction with an initial retrieval step [14]. However, such
approaches are susceptible to mismatches between the (graphemic or phonemic)
ASR output and the audio. Further work extended contextual spelling correction
to incorporate acoustic features [15]; in contrast to that work, we use fast
nearest-neighbors lookup to scale to large correction candidate sets.

Our system builds on the MAESTRO [16] technique for training ASR models in a
self-supervised manner using untranscribed speech and unspoken text. The
suitability of mean- and max-pooled MAESTRO embeddings for retrieval was
explored in [17]; we build on this work by training a retrieval encoder in the
“dual encoder” style [1] which further processes MAESTRO outputs to produce
embeddings that increase recall. Note that, while we build on MAESTRO in this
work, our technique can be extended to any underlying model that produces
joint speech-text representations, such as JOIST [18] and STPT [19].

Our retrieval-based architecture also resembles general keyword spotting [20,
21, 22, 23]. Like the approach of Sacchi et al [24], we use learned acoustic
embeddings and nearest neighbor search to find close matches from an open
vocabulary, but our approach differs in that we use a modality-matching model
to directly match text correction candidates against the utterance audio;
notably, we are able to do direct audio-to-text matching without using text-
to-speech (TTS) synthesis, as is done in [25].

##  3 System Design

###  3.1 Feature extraction with a pretrained MAESTRO model

Our system, illustrated in Fig. 1, performs speech-text matching, where the
mode of the query (speech) is different from that of the retrievable entity
(text). This approach requires a speech-text model that can accurately map
spoken text with acoustic variations (in noise level, pitch, duration, etc.)
to the ground truth text. We use a MAESTRO model that has has been pretrained
on large amounts of both paired and unpaired text-and-speech data to produce
speech-text embedding sequences [16]. For our experiments, the underlying
MAESTRO model was trained on Librispeech [26]. Fig. 2 illustrates how the
MAESTRO model is trained in a semi-supervised manner.

By design, MAESTRO’s neural model is trained on several tasks. The text
encoder contains a graphemes-to-phonemes converter, along with a phonemes-
upsampler model trained on a duration prediction task using phoneme alignments
from an RNN-T decoder. For text inputs, the length of the shared encoder
output embedding sequence is based on the duration prediction task, whereas
the sequence length for speech inputs is proportional to the audio duration.
The speech encoder contains a pretrained w2v-BERT model [27]. A refiner is
used to process the upsampled text inputs during a consistency loss (mean
squared error) minimization task between the speech encoder and refiner output
for paired data.

MAESTRO contains a shared encoder that accepts as input either speech or text
embedding outputs from the corresponding consistency-loss-optimized encoder.
The shared encoder—which comprises the final 12 layers out of 24 from a
pretrained E2E conformer—outputs to an RNN-T decoder. An additional task
involves training the overall architecture on an RNN-T loss function. The
total parameter count for the MAESTRO model is 650 million. Of those
parameters, 300 million are for the speech model, 300 million are for the
shared encoder, and 50 million are for the text model.

Fig. 2: MAESTRO model semi-supervised training process. Fig. 3: Retrieval
Encoder supervised training process.

###  3.2 Speech-text embeddings retrieval with a trained encoder

Our system freezes the MAESTRO shared encoder, using it to output speech-text
embedding sequences that can be processed for similar phrase retrieval via a
retrieval encoder. The retrieval encoder mean pools the sequence of embeddings
across time and then processes the 1024-dimensional mean-pooled output with a
single layer feed forward neural network (FFNN) combined with a ReLU
activation with dropout probability 0.1. The resulting 1024-dimensional output
embedding represents the phrase. The FFNN contains a total of 2.1 million
parameters. During preliminary analysis, omitted for brevity, we observed that
this shallow architecture achieved better retrieval performance than more-
complex architectures. In Section 5, we demonstrate better performance using a
retrieval encoder, rather than simple mean-pool-only representations, for
retrieval.

The retrieval encoder was trained using the dual encoder framework as follows
(illustrated in Fig. 3). Consider a batch of $B$ (text, speech) pairs,
$(t_{i},s_{i})$, $i\in[1,B]$. Let $z(x)=r(\bar{y}(x))$ represent the retrieval
encoder output for text or speech input
$x\in\\{t_{i},...,t_{B},s_{i},...,s_{B}\\}$, where $\bar{y}(x)$ is the mean-
pooled shared encoder output on text or speech input $x$.

The encoder is trained with softmax cross entropy loss:

$-\frac{1}{B}\sum_{i=1}^{B}\log{S_{i}}\textrm{,}$

where $S_{i}=e^{p_{i,i}}/\sum_{j=1}^{B}e^{p_{i,j}}$ is the softmax function
applied to a row in a matrix of speech-text cosine similarities, $p$, such
that

$p_{i,j}=\text{sim}(t_{i},s_{j})=\frac{z(t_{i})\cdot
z(s_{j})}{\|z(t_{i})\|\|z(s_{j})\|}\textrm{.}$

The training data for the retrieval encoder consists of multimodal text-and-
speech embedding pairs originating from the LibriTTS data sets [28], along
with an internal collection of 254K manually-transcribed, anonymous short-form
human audio utterances of random places, businesses, and voice assistant
commands. Together, these pairs formed a training set of 454K examples.

The retrieval encoder was trained on batches with 128 examples over 1.5
million training steps using mini-batch gradient descent with momentum
(momentum parameter of 0.9) [29] with a learning rate of 0.001 without decay
and 10K warm up steps with an initial learning rate of 0.0 and a linear
increase to the final learning rate.

###  3.3 Inference with the shared encoder and retrieval encoder

In our system, the shared encoder and retrieval encoder are used to process
the query audio $s$ and a collection of $M$ contextually-relevant phrases
$t\in{t_{1},...,t_{M}}$, where the embeddings $\\{z(t_{l})\\}_{l=1}^{M}$ are
computed offline. Let $\hat{t}=\operatorname*{arg\,max}_{t}\text{sim}(t,s)$
represent the nearest neighbor text phrase returned during inference.

Our system adds $\hat{t}$ to the n-best list with cost $c(\hat{t})$ as a
function of the E2E cost of the top hypothesis in the original n-best list
$c_{\text{orig}}$, a rewriting aggressiveness hyperparameter $\delta\in[0,1]$,
and the speech-text similarity as follows:

$c(\hat{t})=c_{\text{orig}}+\left(1-\text{sim}\left(\hat{t},s\right)\right)-\delta$
(1)

The cost $c(\hat{t})$ can be interpreted as a pseudo negative log-likelihood.
Ultimately, if $c(\hat{t})<c_{\text{orig}}\Leftrightarrow
1-\text{sim}\left(\hat{t},s\right)<\delta$, then $\hat{t}$ becomes the new top
ASR hypothesis. We limited $\delta$ to the range $[0,1]$ since $\delta\geq 1$
would allow new top hypotheses $\hat{t}$ dissimilar to $s$, i.e.,
$\text{sim}(\hat{t},s)<0$.

While it is possible to union $\hat{t}$ (containing score $c(\hat{t})$) with
the original n-best list and then rescore/rerank the expanded hypothesis list
with an external LM, we did not observe any meaningful improvement in separate
experiments.

##  4 Experiments

###  4.1 Base ASR architecture

Our E2E ASR system is based on the architecture described in [30]. The input
contains 128-dimensional log-Mel features with a 16-dimensional one-hot
domain-id vector appended. The network uses an embedding decoder in which the
previous 2 output tokens are used to look up an embedding before applying a
projection. A joint layer computes the distribution over output tokens based
on the projected output from the embedding decoder network and the encoder
output. The encoder consists of 12 Conformer layers, each of which contains an
8-head self-attention layer and a convolutional kernel with size 15. The
overall network (120 million total parameters) predicts tokens from a
4,096-wordpiece vocabulary [31]. The model was trained using the HAT
factorization [32]; inference incorporates a first-pass LM (5-gram model with
4 million word vocabulary) implemented by a WFST whose input tape contains
wordpieces and output tape contains written-domain words [33]. 400K hours of
anonymized data were used as the training data. When dealing with user data,
our work abides by Google AI principles [34].

###  4.2 Description of models

We compared the following ASR setups in our experiments:

Base corresponds to the base E2E ASR described in Section 4.1. Each of the
setups below corrects outputs from the base ASR.

Mean Pool omits the retrieval encoder and computes $z(x)=\bar{y}(x)\textrm{,
}\forall x\in\\{s,t\\}$ during inference and offline computation of
$\\{z(t_{l})\\}_{l=1}^{M}$ using only the MAESTRO encoders described in
Section 3.1. This setup is identical to that used in previous MAESTRO
retrieval experiments [17].

Mean Pool+Ret (our system) is based on the system described in Sections 3.2
and 3.3.

Text+Ret uses a dual encoder that is trained on (transcript truth, top ASR
hypothesis) pairs. As both inputs are text, this setup uses a single shared
encoder with a two-layer Transformer architecture. Initially, each phrase is
converted to a sequence of character token embeddings (maximum length 100,
each 100-dimensional). In each Transformer layer, the input sequence is
processed by 4 self-attention heads, which together output a sequence of
400-dimensional embeddings. Each embedding is identically processed by a
single layer FFNN that outputs a sequence of 400-dimensional embeddings,
forming the Transformer layer output.

The final embedding is equal to the mean pool applied to the final Transformer
layer output followed by a projection to 1024 dimensions. This architecture
(30.5 million total parameters) is similar to that used in previous text-only
neural matching experiments [3].

###  4.3 Building the embedding database

We sampled $M$=128K transcripts from a base collection of labeled utterances
contextually relevant for voice search, including acquired human audio voice
search queries for directions to locations and TTS queries for application
control, dictation, calling contacts, and media synthesized from grammars. The
transcript text was then encoded using the retrieval encoder during addition
to the embedding database. The offline job in Figure 1 illustrates this
procedure.

###  4.4 Test sets

Throughout our experiments, we evaluate on two types of test sets, each with
dev and eval splits. The first is an in-context (IC) test set, where each
transcript truth is present in the embedding database as a retrievable phrase.
The purpose of this test set is to measure the recall of an ASR correction
system. The IC set contained 4K utterances from the base collection described
in Section 4.3.

The second is an anti test set, which is intended to measure precision. This
test set is designed to represent background voice search queries, i.e., those
to which no retrievable correction applies. Transcript truths from these
utterances are not added to the embedding database. For a contextual ASR
system, performance on such queries ideally should be no worse than the Base
model. AntiTTS contained 5K TTS-synthesized popular generic voice assistant
queries. In addition, AntiH (used during eval split only) contained 9K
anonymized human audio voice search queries.

All test sets contain short-form utterances of a few words reflective of
common voice search use cases, e.g., “places to eat near me,” “what’s the
weather on Friday?”

##  5 Results

We evaluated on the eval split utterances while increasing $M$ to measure the
quality of embeddings when searching against increasingly-large embedding
databases. For each model $v\in\\{\text{Mean Pool},\text{Mean
Pool+Ret},\text{Text+Ret}\\}$, we swept the model-specific rewriting
aggressiveness $\delta_{v}^{*}\in[0,1]$ to find the value producing the lowest
weighted WER score on the IC and AntiTTS dev splits for $M$=8K, with selected
weights: 95% anti, 5% in-context. The dev-split-determined hyperparameter
values for different were then used when evaluating on the eval split.

The results for varying $M$ are shown in Table 1. For the $M\in\\{$8K, 16K,
32K, 64K$\\}$ settings, MeanPool+Ret achieved the lowest in-context WER
without increasing anti set WER relative to Base. On the other hand,
Text+Ret’s AntiTTS WER increased with $M$, suggesting reduced precision likely
due to querying using embeddings of ASR hypotheses that were phonetically
dissimilar to the ground truth, but confusable with the ASR correction
candidates in the embedding database. For all models, WER on AntiH did not
increase relative to the Base AntiH WER. This suggests that our embedding
databases were more likely to contain phrases from domains different from
those in AntiH’s utterances.

Table 1: Eval WER values for increasing embedding database sizes.

Model | Test set |  Embedding database size $M$
---|---|---
8K | 16K | 32K | 64K | 128K
Base | IC | 14.9 | 14.9 | 14.9 | 14.9 | 14.9
AntiTTS | 2.4 | 2.4 | 2.4 | 2.4 | 2.4
| AntiH | 6.2 | 6.2 | 6.2 | 6.2 | 6.2
Text+Ret | IC | 13.9 | 13.9 | 13.9 | 13.9 | 13.9
AntiTTS | 2.4 | 2.5 | 2.6 | 2.7 | 2.9
| AntiH | 6.2 | 6.2 | 6.2 | 6.2 | 6.2
MeanPool | IC | 14.7 | 14.7 | 14.7 | 14.7 | 14.7
AntiTTS | 2.4 | 2.4 | 2.4 | 2.4 | 2.4
| AntiH | 6.2 | 6.2 | 6.2 | 6.2 | 6.2
MeanPool+Ret | IC | 13.6 | 13.7 | 13.8 | 13.8 | 14.0
AntiTTS | 2.4 | 2.4 | 2.4 | 2.4 | 2.4
| AntiH | 6.2 | 6.2 | 6.2 | 6.2 | 6.2

##  6 Conclusion

We introduced a system where similar phrases are retrieved via a nearest
neighbor search with speech-text embeddings to match the utterance audio with
candidate text phrases. This enabled ASR correction with greater precision
than text-based retrieval, which may introduce implausible phrases due to
inputs that are too phonetically dissimilar from the ground truth. We achieved
a relative word error rate reduction of 6% on a voice search test set
containing transcript truths that are included in a database of 128K
retrievable phrases without degrading recognition of general utterances.

##  7 Acknowledgements

The authors would like to thank Pavel Golik, Richard Rose, and Parisa Haghgani
for their insightful comments and discussions.

## References

  * [1] D. Gillick, S. Kulkarni, L. Lansing, A. Presta, J. Baldridge, E. Ie, and D. Garcia-Olano,  “Learning dense representations for entity retrieval,”  in Conference on Computational Natural Language Learning (CoNLL), 2019.
  * [2] R. Guo, P. Sun, E. Lindgren, Q. Geng, D. Simcha, F. Chern, and Sanjiv Kumar,  “Accelerating large-scale inference with anisotropic vector quantization,”  in International Conference on Machine Learning (ICML), 2020.
  * [3] C. Li, P. Rondon, D. Caseiro, L. Velikovich, X. Velez, and P. Aleksic,  “Improving entity recall in automatic speech recognition with neural embeddings,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
  * [4] P. Aleksic, M. Ghodsi, A. Michaely, C. Allauzen, K. Hall, B. Roark, D. Rybach, and Pedro Moreno,  “Bringing contextual information to Google speech recognition,”  in Interspeech, 2015.
  * [5] I. Williams and P. Aleksic,  “Rescoring-aware beam search for reduced search errors in contextual automatic speech recognition,”  in Interspeech, 2017.
  * [6] D. Zhao, T. Sainath, D. Rybach, P. Rondon, D. Bhatia, B. Li, and Ruoming Pang,  “Shallow-fusion end-to-end contextual biasing,”  in Interspeech, 2019.
  * [7] G. Pundak, T. Sainath, R. Prabhavalkar, A. Kannan, and D. Zhao,  “Deep context: End-to-end contextual speech recognition,”  in SLT, 2018.
  * [8] T. Munkhdalai, K.C. Sim, Angad Chandorkar, F. Gao, M. Chua, T. Strohman, and Françoise Beaufays,  “Fast contextual adaptation with neural associative memory for On-Device personalized speech recognition,”  in ICASSP, 2022.
  * [9] F. Chang, J. Liu, M. Radfar, A. Mouchtaris, M. Omologo, A. Rastrow, and S. Kunzmann,  “Context-aware transformer transducer for speech recognition,”  in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021.
  * [10] U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis,  “Generalization through memorization: Nearest neighbor language models,”  in ICLR, 2020.
  * [11] J. Serrino, L. Velikovich, P. Aleksic, and C. Allauzen,  “Contextual recovery of out-of-lattice named entities in automatic speech recognition,”  in Interspeech, 2019.
  * [12] R. Huang, O. Abdel-hamid, X. Li, and G. Evermann,  “Class LM and word mapping for contextual biasing in end-to-end ASR,”  in Interspeech, 2020.
  * [13] X. Wang, Y. Liu, J. Li, V. Miljanic, S. Zhao, and H. Khalil,  “Towards contextual spelling correction for customization of end-to-end speech recognition systems,” 2022.
  * [14] A. Antonova, E. Bakhturina, and B. Ginsburg,  “SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings,”  in Interspeech, 2023.
  * [15] X. Wang, Y. Liu, J. Li, and S. Zhao,  “Improving contextual spelling correction by external acoustics attention and semantic aware data augmentation,”  in ICASSP, 2023.
  * [16] Z. Chen, Y. Zhang, A. Rosenberg, B. Ramabhadran, P. Moreno, A. Bapna, and H. Zen,  “MAESTRO: Matched speech text representations through modality matching,”  in Interspeech, 2022.
  * [17] G. Wang, K. Kastner, A. Bapna, Z. Chen, A. Rosenberg, B. Ramabhadran, and Y. Zhang,  “Understanding shared speech-text representations,”  in ICASSP, 2023.
  * [18] T. Sainath, R. Prabhavalkar, A. Bapna, Y. Zhang, Z. Huo, Z. Chen, B. Li, W. Wang, and T. Strohman,  “JOIST: A joint speech and text streaming model for ASR,”  in SLT, 2022.
  * [19] Y. Tang, H. Gong, N. Dong, C. Wang, W. Hsu, J. Gu, A. Baevski, X. Li, A. Mohamed, M. Auli, and J. Pino,  “Unified speech-text pre-training for speech translation and recognition,”  in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
  * [20] M. J. F. Gales, K. M. Knill, A. Ragni, and S. P. Rath,  “Speech recognition and keyword spotting for low resource langauges: Babel project research at CUED,”  in International Workshop on Spoken Language Technologies for Under-Resourced Languages, 2014.
  * [21] K. Audhkhasi, A. Rosenberg, A. Sethy, B. Ramabhadran, and B. Kingsbury,  “End-to-end ASR-free keyword search from speech,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1351–1359, 2017.
  * [22] T. Sainath and C. Parada,  “Convolutional neural networks for small-footprint keyword spotting,”  in Interspeech, 2015.
  * [23] R. Alvarez and H.-J. Park,  “End-to-end streaming keyword spotting,”  in ICASSP, 2019.
  * [24] N. Sacchi, A. Nanchen, M. Jaggi, and M. Cernak,  “Open-Vocabulary Keyword Spotting with Audio and Text Embeddings,”  in Interspeech.
  * [25] D.M. Chan, S. Ghosh, A. Rastrow, and B. Hoffmeister,  “Domain adaptation with external off-policy acoustic catalogs for scalable contextual end-to-end automated speech recognition,”  in ICASSP, 2023.
  * [26] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in ICASSP, 2015.
  * [27] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Yonghui Wu,  “w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  in ASRU, 2021.
  * [28] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu,  “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,”  in Interspeech, 2019.
  * [29] I. Sutskever, J. Martens, G. Dahl, and G. Hinton,  “On the importance of initialization and momentum in deep learning,”  in ICML, 2013.
  * [30] T. Sainath, Y. He, A. Narayanan, R. Botros, R. Pang, D. Rybach, C. Allauzen, E. Variani, J. Qin, Q.-N. Le-The, A. Gruenstein, A. Gulati, B. Li, C. Peyser, C.-C. Chiu, D. Caseiro, E. Guzman, I. McGraw, J. Yu, M.D. Riley, P. Rondon, Q. Liang, S. Mavandadi, S.-Y. Chang, T. Strohman, W. R. Huang, W. Li, Y. Wu, and Y. Zhang,  “An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,”  in Interspeech, 2021.
  * [31] M. Schuster and K. Nakajima,  “Japanese and Korean voice search,”  ICASSP, 2012.
  * [32] E. Variani, D. Rybach, C. Allauzen, and M. Riley,  “Hybrid autoregressive transducer (hat),”  in ICASSP, 2020.
  * [33] C. Allauzen, E. Variani, M. Riley, D. Rybach, and H. Zhang,  “A hybrid seq-2-seq ASR design for on-device and server applications,”  in Interspeech, 2021.
  * [34] Google,  “Artificial intelligence at Google: Our principles,” https://ai.google/principles.