# Design and Manufacture of Flexible Epidermal NFC Device for Electrochemical
Sensing of Sweat
††thanks: Work funded by Regione Lazio, project E-CROME (biosEnsori su Carta
wiReless per la telemedicina in Oncologia e la misura di eMocromo ed
Elettroliti; Development of NFC interface sensors for the measurement of
biomarkers in blood), CUP: E85F21001040002.

Adina Bianca Barba  Pervasive Electromagnetics Lab
University of Rome Tor Vergata
Rome, Italy
<EMAIL_ADDRESS>Giulio Maria Bianco  Pervasive
Electromagnetics Lab
University of Rome Tor Vergata
Rome, Italy
<EMAIL_ADDRESS>Luca Fiore  NanoBioSensing Lab
University of Rome Tor Vergata
Rome, Italy
<EMAIL_ADDRESS>Fabiana Arduini  NanoBioSensing Lab
University of Rome Tor Vergata
Rome, Italy
<EMAIL_ADDRESS>Gaetano Marrocco  Pervasive Electromagnetics Lab
University of Rome Tor Vergata
Rome, Italy
<EMAIL_ADDRESS>Cecilia Occhiuzzi  Pervasive Electromagnetics
Lab
University of Rome Tor Vergata
Rome, Italy
<EMAIL_ADDRESS>
###### Abstract

Flexible and epidermal sensing devices are becoming vital to enable precision
medicine and telemonitoring systems. The NFC (Near Field Communication)
protocol is also becoming increasingly important for this application since it
is embedded in most smartphones that can be used as pervasive and low-cost
readers. Furthermore, the responder can be passive and can harvest enough
power to perform electromagnetic sensing. Finally, the NFC coils are robust to
bending and to the human body’s presence. This contribution details the design
of a new flexible device, including an electrochemical sensor communicating
through the NFC protocol. A spiral NFC antenna is designed, and a manufactured
prototype is experimentally tested to quantify the robustness to the inter-
wearer variability and the bending. Lastly, the sensory data retrieval is
validated by comparison with a portable potentiostat. The realized sensor can
be comfortably worn and be easily read by smartphones independently from the
wearer and from the point of application and could be used in future for
estimating the user’s psycho-physical health by analyzing the body’s sweat.

###### Index Terms:

Biosensor, chemical sensor, flexible device, Healthcare Internet of Things,
Near Field Communication, epidermal electronics.

##  I Introduction

Smart wearable and epidermal sensors are currently being investigated for
healthcare applications following the paradigm of precision medicine [1, 2,
3]. As an example, many devices for sensing biosignals like skin temperature
and coughing [4] or analytes in the sweat [5] have been recently proposed.
However, the wireless communication of the sensed data is still a major
concern together with the use of local power supply, data security and the use
of low-cost readers for preserving user comfort and widespread diffusion [6,
7]. The Near Field Communication (NFC) protocol is an excellent option since
the short reading distances can protect the privacy of the patient [8].
Moreover, most smartphones can read NFC devices that are nowadays a pervasive
technology and will become even more utilized when interoperability issues
will have been overcome [9]. The HF (high frequency) spiral antennas of NFC
tags [10, 11] are remarkably robust to bending [12, 13] and the environment
[14], enabling integration in flexible sensors where limited space is
available [15].

|
---|---
(a) | (b)
Figure 1: (a) Concept of sweat monitoring system comprising a passive,
flexible and soft electrochemical sensor attached to the skin and capable of
wirelessly communicating with a smartphone (b) Numerical simulations of the
designed NFC device. Exploded view of the model of the device.

The electrochemical sensors using commercial-off-the-shelves components
communicating through NFC were proven extremely promising, mainly thanks to
the highly efficient energy harvesting capability, which allows even passive
microchips to perform complex chemical measurements such as amperometry and
voltammetry [16, 17]. This contribution hence details the design and test of a
new epidermal biosensor employing the NFC protocol [Fig. 1(a)]. The device is
capable of performing real-time analysis of sweat and particularly of
cortisol. Indeed, the cortisol levels can be linked to high levels of stress
[18] and could enable precision monitoring, especially during sports activity
or for workers during a hectic activity like soldiers or medical staff in
hospital wards during the COVID-19 pandemic [4]. The device is totally passive
and can be interrogated by smartphones.

##  II Design of the Flexible NFC Responder

The wireless epidermal device is composed of a spiral antenna, an NFC
microchip, and an electrochemical sensor [Fig. 1(b)]. The design followed
three main requirements: minimized size and weight, flexibility and
biocompatibility of materials, reduced electromagnetic interference during
interrogation. Therefore, the device has a rectangular shape whose overall
size is similar to common skin patches. The antenna is located on the left
side, far from the IC (integrated circuit) and the sensing circuitry, to
reduce eventual detrimental effects resulting from the magnetic interaction
with the smartphone [19]. The antenna is made of copper micro-wires (diameter
approximately 100 $\mu$m) directly laid down on a medical-grade self-adhesive
membrane (Tegaderm [20]). IC, tuning, and sensing circuitry are placed on a
flexible PCB. Finally, the sensor is located on the right side and is provided
with a plug&play connector to enable multiple usages. The NFC SIC $4341$ (by
Silicon Craft Technology) is the selected IC that is compatible with
electrochemical sensing and can be read by a smartphone.

NFC tag antenna can be modelled as an inductance $L$ having loss resistance
$R$, and a tuning capacitance $C_{tun}$, while the IC as a capacitance-
resistance parallel ($C_{chip}$ and $R_{chip}$). The NFC tag must also have a
quality factor $Q=R_{chip}\left[2\pi f_{r}L\right]^{-1}$ lower than a maximum
value $Q_{max}$ to channel all the frequencies contained in the HF-RFID
spectrum. For targets employing the ISO/IEC $14443$A protocol like the
considered one, $Q_{max}=25$ is a commonly accepted value [21]. By neglecting
the parasitic capacitances, the resonance frequency $f_{r}$ of the equivalent
circuit can be evaluated as
$f_{r}\simeq\left[2\pi\sqrt{L\left(C_{tun}+C_{chip}\right)}\right]^{-1}$ [14].
Even if the NFC protocol employs the carrier frequency of $13.56$ MHz, for the
design, a higher $f_{r}=15.7$ MHz is considered to account for the frequency
shift caused by the proximity of the reader’s antenna during communications
[16]. Since $C_{chip}=50$ pF for the SIC $4341$ IC, by imposing $C_{tun}=27$
pF the resulting inductance is $L=1.36$ $\mu$H.

The NFC tag was designed through electromagnetic simulations111By CST
Microwave Studio $2018$, frequency domain.. A rectangular coil made of five
turns of a thin copper wire is chosen as the antenna’s geometry. The IC,
instead, is placed on a pad (size $20$ mm $\times$ $15$ mm $\times$ $0.8$ mm)
made of Kapton (modeled as a lossless dielectric with relative permittivity
$3.3$ [22]), and it is soldered to the antenna and to a connector through
$35$-$\mu$m-thick copper traces [Fig. 1(b)]. To account for the presence of
the human body, the tag is centered on a multi-layer phantom having a square
section $20$ cm $\times$ $20$ cm and composed of dry skin thick $1$ mm, fat
thick $1$ cm, and muscle thick $4$ cm (electrical properties at $f=15.7$ MHz
from [23]).

To achieve the needed inductance, four parameters were varied: the rectangle’s
longer side $L_{x}$, the rectangle’s shorter side $L_{y}$, the space between
the wires $s$, and the radius of the wires $w$. The parameters’ values
obtained by numerical simulations are resumed in the caption of Fig. 3(a), and
they return, at $f=15.7$ MHz, $L=1.36$ $\mu$H and $Q=18<25$ (Fig. 2). The
total size $L\times W$ of the device allows for multiple positioning over the
body while preserving the overall conformability and wearer comfort.

Figure 2: $L$ and $Q$ vs $f$ evaluated by numerical simulations. The design frequency of $15.7$ MHz is highlighted by a vertical line. |
---|---
(a) | (b)
Figure 3: (a) Prototype of the designed flexible device. Parameters $W=31.5$
mm, $L=65$ mm, $L_{x}=30$ mm, $s=0.9$ mm $L_{y}=21.7$ mm, $w=40$ $\mu$m (b)
Example of the flexed device.

##  III Manufacturing and Measurements

###  III-A Manufacturing

For the sake of prototyping, the copper traces are obtained through a milling
machine on an FR-$4$ board instead of the Kapton [24]. The spiral coils made
of thin, flexible and moderately stretchable [25] copper wire are placed by
machines on the Tegaderm 3M medical plaster [26]. Then, the coil and the
traces are manually soldered together afterwards. The overall epidermal
biosensor is entirely deployed on the transparent and breathable medical
plaster [Fig. 3(a)]. Even though the FR-$4$ is rigid, the overall sensor is
highly flexible for most of the space being occupied by the spiral made of
wires, and it can be comfortably worn all over the body [Fig. 3(b)].

###  III-B Communication test

The prototype is tested with the Tagformance Pro (C$60$ antenna) both in the
air reference condition (namely, when the human body is absent) and when it is
placed on the arm of eight different wearers (six females, two males; mean
body mass index $22.7\pm 4.2$). The reading ranges and the quality of the
magnetic coupling, which is expressed by the passive load modulation [27], are
depicted in Fig. 4. The device can be read up to a distance of $40$ mm. The
target responds slightly better on the human body than in the air ($+10\%$
reading range) since it was designed on a body phantom. The inter-subject
variability [28] yields a standard deviation as low as $1.2$ mm. Similarly,
the prototype is rather robust to the bending, and the maximum communication
range is reduced by just $3$ mm when a bend angle of $1.5$ rad ($\sim 85^{o}$
degrees) is applied by attaching the sensor to an empty plastic cylinder
having a known radius (Fig. 5). Accordingly, a typical NFC range of about $3$
cm is expected independently from the user and the point of application of the
sensor. In all the experiments, the coupling’s quality follows the range
variations, being remarkably stable and depending primarily on the reader-tag
distance.

Figure 4: Reading distances and load modulation when the biosensor is placed
in air and when it is worn by eight different subjects. The ”on-body” column
shows the average over the eight wearers. Measurement setup at the bottom.
Figure 5: Load modulation, reading distances, and the corresponding reduction
of communication range when the non-worn tag is bent for different bending
angles. Measurement setup at the bottom.

###  III-C Sensing test

The capability of the NFC microchip of retrieving the data from the given
sensor was finally assessed [16]. The considered sensor of cortisol was
developed starting from the chemical probe in [29, 7], and, if wet, it returns
an electrical current inversely proportional to the cortisol concentration.
The current can be measured through chronoamperometry, i.e. by imposing a
transient voltage and consequently measuring the current during a well-
specified time interval.

For comparison, the sensor’s output current is also measured with a portable
potentiostat (Emstat Blue, by PalmSens). The NFC sensor is instead
interrogated by an Android smartphone through the Chemister app by Silicon
Craft.

Figure 6: Electric current vs time curve when the biosensor is soaked by a
cortisol concentration of $40$ ng/mL. In the inset: concept of the biosensory
data retrieval by the Chemister app.

The current vs time curves of the sensor produced by a concentration of
cortisol of $40$ ng/mL are shown in Fig. 6. The two curves are in close
agreement and return the same current at their stationary states ($120$ s).
Thus, the selected IC is viable for electrochemical sensing through the
designed flexible device and smartphones.

##  IV Conclusion

A flexible epidermal sensing device communicating through the NFC protocol was
designed and tested for application to cortisol sensing in sweat. The
prototype is highly robust to the inter-user variability and bending thanks to
the HF communications so that it achieved a reading range of about $3.5/4$ cm
for every considered user and point of application on the body. The
preliminary sensing test validates the accuracy of the retrieved sensory data,
which is comparable with that of much more expensive bench equipment. A
miniaturized and fully flexible prototype is currently under development and
will be presented during the Conference together with a complete test campaign
in real conditions aimed at validating the sensing capabilities of the device
over multiple subjects.

## References

  * [1] Y. Yu, H. Y. Y. Nyen, W. Gao, and A. Javey, “Flexible electrochemical bioelectronics The rise of in situ bioanalysis,” _Advanced Materials_ , vol. 32, pp. 1–25, 2021.
  * [2] I. Jeong, D. Bychklov, and P. C. Searson, “Wearable devices for precision medicine and health state monitoring,” _IEEE Trans. Biomed. Eng._ , vol. 66, no. 5, pp. 1242–1258, May 2019.
  * [3] B. W. Amin _et al._ , “Smart sensor systems for wearable electronic devices,” _Polymers_ , vol. 9, no. 8, pp. 1–41, Jul. 2017.
  * [4] G. M. Bianco, C. Occhiuzzi, N. Panunzio, and G. Marrocco, “A survey on radio frequency identification as a scalable technology to face pandemics,” _IEEE Journal of Radio Frequency Identification_ , vol. 6, pp. 77–96, Jan. 2022.
  * [5] S. Jo, D. Sung, S. Kim, and J. Koo, “A review of wearable biosensors for sweat analysis,” _Biomedical Eng. Lett._ , vol. 11, pp. 177–129, May 2021.
  * [6] S. Pillai, A. Upadhayay, D. Sayson, B. H. Nguyen, and S. D. Tran, “Advances in medical wearable biosensors: Design, fabrication and materials strategies in healthcare monitoring,” _Molecules_ , pp. 1–29, Dec. 2021.
  * [7] V. Mazzaracchio, L. Fiore, S. Nappi, G. Marrocco, and F. Arduini, “Medium-distance affordable, flexible and wireless epidermal sensor for ph monitoring in sweat,” _Talanta_ , pp. 1–10, Jan. 2021.
  * [8] F. Di Rienzo, A. Virdis, C. Vallati, N. Carbonaro, and A. Tognatti, “Evaluation of NFC-enabled devices for heterogeneous wearable biomedical application,” _IEEE J. Radio Frequency Identification_ , vol. 4, no. 4, pp. 373–383, Dec. 2020.
  * [9] M. Erb, C. Steger, M. Troyer, and J. Preishuber-Pfluegl, “Towards fully interoperable NFC devices,” in _IEEE International Conference on RFID_ , 2020, pp. 1–8.
  * [10] I.-F. Chen, C.-M. Peng, and Z.-D. Yan, “A simple NFC RF performance measurement method based on ISO/IEC 14443 standard,” _IEEE J. Radio Frequency Identification_ , vol. 4, no. 4, pp. 438–443, 2020.
  * [11] R. Del-Rio-Ruiz, J.-M. Lopez-Garde, J. L. Macon, and H. Rogier, “Design and performance analysis of a purely textile spiral antenna for on-body NFC applications,” in _IEEE MTT-S Int. Microw. Workshop Series Advanced Materials Processes RF THz Appl._ , 2017, pp. 1–3.
  * [12] Y. Jian _et al._ , “e-Textile embroidered wearable near-field communication RFID antennas,” _IET Microwav., Antennas & Propag._, vol. 1, pp. 99–104, Sep. 2018.
  * [13] V. Le, U. Lemmer, and E. Mackensen, “Analysis of miniaturized printed flexible RFID/NFC antennas using different carrier substrates,” _IEEE Journal of Radio Frequency Identification_ , vol. 4, no. 4, pp. 428–437, 2020.
  * [14] A. Lazaro, R. Villarino, and D. Girbau, “A survey of NFC sensors based on energy harvesting for IoT applications,” _Sensors_ , vol. 18, no. 11, pp. 1–26, Nov. 2018.
  * [15] R. Iqbal, M. D. Saeed, and A. Ahmad, “Wearable Near Field Communication ring antenna for mobile communication,” in _Int. Conf. Open Source Sys. Technol._ , 2014, pp. 146–150.
  * [16] K. Krorakai, S. Klangphukhiew, S. Kulchat, and R. Patramanon, “Smartphone-based NFC potentiostat for wireless electrochemical sensing,” _Applied Sci._ , pp. 1–13, Jan. 2021.
  * [17] L. Xu _et al._ , “Characterization and modeling of embroidered NFC coil antennas for wearable applications,” _IEEE Sensors J_ , vol. 20, no. 23, pp. 14 501–14 513, Dec. 2020.
  * [18] M. Cay _et al._ , “Effect of increase in cortisol level due to stress in healthy young individuals on dynamic and static balance scores,” _Northern clinics of Istanbul_ , vol. 5, no. 4, p. 295, 2018.
  * [19] C. Nalbantoǧlu, T. Kiehl, R. God, T. Stadtler, R. Kebel, and R. Bienert, “Electromagnetic compatibility (EMC) for integration and use of near field communication (NFC) in aircraft,” in _ESA Workshop on Aerospace EMC_ , Valencia, Spain, May 2016, pp. 1–6.
  * [20] 3MTM, “3MTM TegadermTM Transparent Film Dressing Frame Style,” https://www.3m.com/3M/en_US/p/d/b00035590/, 2022.
  * [21] A. Lazaro, B. Marti, R. Villarino, and D. Girbau, “NFC sensors based on energy harvesting for IoT applications,” in _Recent Wireless Power Transfer Technologies_ , 1st ed. London, United Kingdom: InTechOpen, 2019, ch. 2, pp. 23–43.
  * [22] J. O. Simpson and A. K. St. Clair, “Fundamental insight on developing low dielectric constant polyimides,” _Thin Solid Films_ , pp. 480–485, 1997\.
  * [23] Italian national research council - Institute for Applied Physics, “Calculation of the dielectric properties of body tissues,” http://niremf.ifac.cnr.it/tissprop/htmlclie/htmlclie.php, 1997, (accessed Feb. 08, 2022).
  * [24] Mipec, “PCB milling machine 4MILL300,” https://www.webshop.mipec.eu/PCB-milling-machine-4MILL300-d70.htm?tab=description.
  * [25] C. Miozzi, G. Diotallevi, M. Cirelli, P. P. Valentini, and G. Marrocco, “Radio-mechanical characterization of epidermal antennas during human gestures,” _IEEE Sensors Journal_ , vol. 20, no. 14, pp. 7588–7594, 2020\.
  * [26] Mecsrtar s.r.l., “Mecstar RF350 - HF & UHF Innovative RFID production,” https://www.youtube.com/watch?v=YZLupD1AR0g, 2016.
  * [27] D. Paret, _Design Constraints for NFC Devices_ , 1st ed. Hoboken, NJ, USA: John Wiley & Sons, 2016.
  * [28] G. M. Bianco, C. Vivarelli, S. Amendola, and G. Marrocco, “Experimentation and calibration of near-field UHF epidermal communication for emerging tactile internet,” in _Proc. 5th Int. Conf. Smart Sustain. Technol._ , Split, Croatia, 2020, pp. 1–4.
  * [29] S. Nappi _et al._ , “A plug&play flexible skin sensor for the wireless monitoring of pandemics,” in _IEEE Int. Conf. Flexible and Printable Sensors Sys._ , Manchester, United Kingdom, 2021, pp. 1–4.


# GPU-Accelerated WFST Beam Search Decoder for CTC-based Speech Recognition

###### Abstract

While Connectionist Temporal Classification (CTC) models deliver state-of-the-
art accuracy in automated speech recognition (ASR) pipelines, their
performance has been limited by CPU-based beam search decoding. We introduce a
GPU-accelerated Weighted Finite State Transducer (WFST) beam search decoder
compatible with current CTC models. It increases pipeline throughput and
decreases latency, supports streaming inference, and also supports advanced
features like utterance-specific word boosting via on-the-fly composition. We
provide pre-built DLPack-based python bindings for ease of use with Python-
based machine learning frameworks at https://github.com/nvidia-riva/riva-
asrlib-decoder. We evaluated our decoder for offline and online scenarios,
demonstrating that it is the fastest beam search decoder for CTC models. In
the offline scenario it achieves up to 7 times more throughput than the
current state-of-the-art CPU decoder and in the online streaming scenario, it
achieves nearly 8 times lower latency, with same or better word error rate.

Index Terms—  Automatic speech recognition, decoder, WFST, parallel computing

##  1 Introduction

Beam search is an algorithm used during inference in ASR as an alternative to
greedy search. Unlike greedy search, which selects the most likely output
token determined by the acoustic model at every time step, beam search
maintains a “beam” of the top K most likely sequences at each step. It not
only to provides results with low error rates, but also allows for integration
of external information, e.g., language models, contextual biasing, etc., that
greedy search cannot account for.

Fig. 1: Timing breakdown of NeMo’s Conformer CTC Large inference pipeline on
Lbirispeech test-clean, using Librispeech’s 3-gram.pruned.3e-7.arpa.gz
language model. Beam search ran on one CPU core, while feature extractor and
acoustic model ran on one A100-80GB PCIe.

While major components of ASR pipelines are already GPU accelerated, e.g.,
feature extraction and acoustic model, current beam search implementations for
CTC models are all CPU-based, consuming the majority of inference runtime,
c.f., figure 1. Even though both feature extraction and acoustic model are
already offloaded to the GPU, beam search on the CPU consumes nearly 90% of
the runtime. While there continue to be software-level innovations to
accelerate (neural network) acoustic model inference on GPU [1], Amdahl’s Law
limits end-to-end speed-up, unless beam search is also accelerated. The
maximum speed-up that can be achieved by further accelerating the GPU part of
the workload is a mere 11%.

Beam search can easily be parallelized over the elements in a batch and run on
multiple CPU cores, alleviating the decoder bottleneck. However, performance
does not scale linearly with the number of cores and the ratio of CPU cores
per GPU in a system puts an upper bound on this approach. To date, there are
no cloud systems with a ratio of CPU cores to GPUs exceeding 16. E.g.,
Nvidia’s DGX A100 pairs 128 CPU cores with 8 A100 GPUs [2] and AWS’s EC2’s
p4d.24xlarge offers only 96 CPU cores with 8 A100 GPUs [3]. The results in
figure 1 were obtained using the Flashlight Decoder on a single core. Even
when enabling 16 CPU cores for decoding it remains the bottleneck consuming
43% of end-to-end runtime. Our proposed GPU decoder (henceforth, “CUDA WFST
Decoder”) alleviates the bottleneck, accelerating decoding (over the 16 CPU
core baseline) by a factor of 15.6 and yielding a 4.6x end-to-send pipeline
speed-up, c.f., section 4.1.

Given that feature extraction and acoustic model are already GPU based, the
end-to-end performance gains from using a GPU decoder directly translate into
cost and power savings, the latter becoming a major concern for data centers.
In other words, accelerating beam search is imperative for cost- and power-
efficient ASR inference.

The rest of the paper is organized as follows. We discuss related work and
then describe algorithm and implementation details. We evaluate offline
transcription performance and word error rate; real-time streaming
transcription throughput and latency; and word boosting’s effect on word error
rate. We summarize our contribution and conclude with describing future work.

##  2 Related work

There are a few GPU-accelerated beam search decoders for ASR. Ours is an
extension of the work described in [4], which supports only hybrid DNN-HMM
models in Kaldi. [5] lacks online decoding or ability to condition on acoustic
model log likelihoods. [6] has a critical error: it tries to synchronize CUDA
thread blocks, which is disallowed by the CUDA programming model; this causes
sporadic crashes. It also does not support streaming inference. K2 [7] also
has a GPU-accelerated WFST-based CTC decoder, but it does not support online
streaming inference or the other extensions we describe. All support n-gram
language models.

The most popular CPU-based decoder today is Flashlight’s [8]. It is a non-WFST
dynamic decoder that uses KenLM and prefix tries. It does support streaming
inference. It is faster than other CPU-based decoders, but unfortunately not
fast enough to keep the GPU fully fed doing acoustic model inference. It does
not support word boosting. Its python bindings also do not support multi-
threading because they do not correctly drop the global interpreter lock while
running the C++-based decoder code. We forked the code base and added proper
multi-threading support for the sake of fair comparison. It is the baseline
implementation we compare to in this paper.

##  3 Algorithm description

Fig. 2: “Normal” $T$ transducer that encodes the CTC transduction rules (that
blank should not be emitted as an output, and that repeating the same token on
the output requires at least one blank input in between). Adapted from [9].
Fig. 3: “Compact” $T$ transducer. It does not encode the CTC rule that two
consecutive symbols that are the same should be transduced to a single symbol.
Adapted from [9].

The exact details of the WFST decoding algorithm are described in detail in
[4]. At a high level, an iteration is run for the log likelihoods output by
the acoustic model at each timestep. There are three stages that run for each
iteration: (1) ExpandArcs to expand the current set of best tokens to a new
set of tokens, (2) Prune to remove tokens outside the “beam”, and (3) D2H Copy
to copy these tokens and backpointers from the device to the host in order to
construct a lattice or least cost path later.

This generic WFST decoder can be used for CTC models simply by creating a $T$
WFST that models the rules of CTC sequence transduction properly. $T$ is then
composed with $L$ and $G$ WFSTs [10]. For more details, see [9]. Like [9], we
investigate topologies that do not perfectly model the CTC transduction rules.
In particular, we investigate the “compact” $T$ WFST topology, finding that it
greatly improves word error rate compared to the correct “normal” $T$ WFST
topology. See figures 2 and 3. For a discussion of the creation of the $TLG$
WFST used for the actual speech decoding, see [10]. Our software library
includes scripts to create these WFSTs.

We also make several improvements to the code in [4] to improve throughput
without affecting the correctness of the algorithm:

  1. 1.

Since the creation of [4], CUDA kernel execution has gotten faster by a factor
of two just from hardware architectural improvements. This has caused kernel
launch overhead to take up to 20% of each iteration of beam search because it
is often the case that the previous kernel has finished long before the CPU
has launched the next kernel. We eliminate this CUDA kernel launch overhead by
using cuda graphs [11].

  2. 2.

The original implementation waited to do the next iteration of beam search
until the previous iteration’s tokens had been copied to CPU. Since device-to-
host latencies have not improved commensurately with GPUs’ architectural
improvements, this takes up about 25% of each iteration. Fortunately, memory
capacity on GPUs have increased by a factor of two or more since then, so we
can use double-buffering to prevent these stalls without worry about memory
usage.

We next discuss the particulars around implementing streaming inference and
word boosting on top of this decoder.

###  3.1 Streaming inference

Effective streaming inference with GPUs requires dynamic batching of incoming
streams of audio. In streaming automatic speech recognition, a new chunk from
a given audio stream comes in at a nearly constant rate, typically at some
multiple of the frame shift of the audio featurizer (frequently, but not
always, 10 ms). Because the latency of doing inference on a given chunk is
effectively always less than the this request rate, we must save the state of
the acoustic model (e.g., the hidden state vectors of an LSTM), the state of
the decoder (i.e., the current active tokens), and the state of the feature
extractor (e.g., the random seed used for dithering and -rarely in modern
architectures- the current ivector value) between requests from the same audio
stream. While that audio stream is waiting for its next chunk, requests from
other ready audio streams can be serviced. Fortunately, saving these states is
effectively done by doing a scatter operation to a memory buffer in GPU
memory. Meanwhile, loading can be done by a gather operation.

To help interpret our results, we make a basic observation from queueing
theory [12]. We can model our inference server as an M/D/1 queue. That is, we
have only one server, which takes a constant amount of time ($\mu=1/D$, the
“serving rate”) to process each request; meanwhile, requests come in following
a Poisson distribution at a rate of $\lambda$. Note that $D$ here refers to
the latency of the server to respond to a request, often referred to as the
“compute latency”. This is not a perfect model because serving rates have
variance and this model assumes an infinite length queue (but computer memory
is finite). A Poisson distribution is a reasonable model for incoming requests
because each request is actually a batch of audio chunks from multiple audio
streams. While the audio from a single stream comes in at an approximately
constant rate, there is a variable wait time because earlier arriving audio
streams must wait until a full batch worth of requests arrives.

In this model, the average total latency to service a request can be modeled
as:

$\omega=\frac{1}{\mu}+\frac{\lambda}{2\mu(\mu-\lambda)}$

The first term corresponds to the time to process a request (compute latency),
while the second term corresponds to the time a request spends waiting (queue
latency). The main take away is that request wait time decreases quadratically
with the serving rate $\mu$, because of the second term. Since serving rate is
just the inverse of the compute latency $D$, we observe that the best way to
decrease total latency for a request to get a response in streaming inference
regimes is to decrease the latency of the inference engine, not necessarily to
increase its throughput by scaling to larger batch sizes.

###  3.2 Word boosting

Fig. 4: Example word boosting WFSA. “Fish” gets a boost of 6 and “fox” gets a
boost of 2. “Phi” stands for a failure arc; if no other label is accepted,
then that arc is taken.

We implement word boosting as follows. The decoder is given the WFST $TLG$
shared among all utterances. Then, when a new audio stream is created, the
creator of it can also pass a word boosting graph $B$ via a table mapping in-
vocabulary word IDs to scores. $B$ is created from the table as a single state
Weighted Finite State Acceptor (WFSA); see figure 4. The decoder will then
decode that specific utterance as if its decoding graph were $TLG\circ B$. Of
course, $TLG$ is not explicitly composed on the right with $B$, since that
would require making a copy of $TLG$ for every utterance requesting word
boosting (which is typically several gigabytes in size). Instead, we take
advantage of the fact that $B$ is a single state to implement it as a hash
table on GPU (in particular, we use cuco::static_map [13]) mapping word IDS to
scores. Every time an arc with a non-epsilon olabel is encountered in $TLG$
during the ExpandArcs step of the algorithm, we always know we will be in
$B$’s one and only state, so we can simply check whether that olabel is in the
cuco::static_map object for that utterance. The pseudo-code is given in figure
5.

1:for token $\in$ main_queue do

2: for arc $\in$ token.state.out_arcs do

3: new_cost $\leftarrow$ token.accumulated_cost + acoustic_cost[arc_id.ilabel]
+ arc.graph_cost

4: if arc_id.olabel $!=0$ then

5: new_cost $\leftarrow$ new_cost + b_fst[arc_id.olabel].graph_cost

6: end if

7: end for

8:end for

Pseudo-code of Expand Arcs operation to support word boosting.  Fig. 5: The
changes to the original algorithm to support word boosting are highlighted in
green. Note that in practice both for loops are run in parallel on the GPU.

Note that, since the decoder is seeking a minimum cost path, “boosted” words
should be given a negative score.

There are a few limitations to this approach.

First, the ideal value for word boosting is dependent upon the beam width.
Suppose that the numerical beam width is $W$, and that the word “cat” has a
boosting score of $W$ as well. Assuming that the log likelihood scores from
the acoustic model and the $TLG$ arc scores are approximately equal at each
time step, sequences containing words other than “cat” will fall out of the
beam, resulting in the final transcript almost always being a sequence of the
word “cat” repeated multiple times.

Second, $TLG\circ B$ will not be “stochastic”. That is, the arcs entering any
given state will not necessarily sum to one in the log semiring. This does not
appear to be an issue in practice. Note that, if following the WFST creation
recipe used by Kaldi [14], $TLG$ will not be stochastic to begin with.

Third, this approach can work poorly if the olabels are not pushed as far left
as possible in $TLG$. One can do this either by construction in the lexicon
following the recipe in [10] or by doing label pushing on the olabels of the
lexicon in [15]. This allows the boost scores to participate in beam search
earlier rather than later. The scripts for graph creation in our open source
software handle this quirk.

Fourth, out-of-vocabulary words will not be boosted because they naturally
won’t compose with the olabels of $TLG$.

##  4 Results

model | dataset | flashlight wer% | cuda compact wer% | cuda normal wer%
---|---|---|---|---
stt_en_conformer_ctc_small | test-clean | 3.28 | 3.17 | 3.56
stt_en_conformer_ctc_small | test-other | 6.99 | 6.83 | 7.12
stt_en_conformer_ctc_medium | test-clean | 2.59 | 2.65 | 3.20
stt_en_conformer_ctc_medium | test-other | 5.43 | 5.32 | 5.73
stt_en_conformer_ctc_large | test-clean | 2.31 | 2.21 | 2.74
stt_en_conformer_ctc_large | test-other | 4.26 | 4.10 | 4.53
Table 1: Offline Word Error Rate Table. Best results per row are bolded. model | dataset | flashlight | cuda compact | cuda normal | compact to flashlight speed-up
---|---|---|---|---|---
stt_en_conformer_ctc_small | test-clean | 646 | 4320 | 4464 | 6.7
stt_en_conformer_ctc_small | test-other | 614 | 4110 | 4528 | 6.7
stt_en_conformer_ctc_medium | test-clean | 549 | 3727 | 3721 | 6.8
stt_en_conformer_ctc_medium | test-other | 536 | 3910 | 4082 | 7.3
stt_en_conformer_ctc_large | test-clean | 488 | 2261 | 2322 | 4.6
stt_en_conformer_ctc_large | test-other | 465 | 2205 | 2189 | 4.7
Table 2: Offline RTFx end-to-end pipeline throughput for flashlight, cuda compact, and cuda normal decoders model | dataset | Flashlight (single core) | Flashlight (16 cores) | cuda compact
---|---|---|---|---
stt_en_conformer_ctc_small | test-clean | 1 | 4.1 | 47.5
stt_en_conformer_ctc_small | test-other | 1 | 4.1 | 55.4
stt_en_conformer_ctc_medium | test-clean | 1 | 4.6 | 67.0
stt_en_conformer_ctc_medium | test-other | 1 | 4.4 | 66.6
stt_en_conformer_ctc_large | test-clean | 1 | 3.4 | 53.2
stt_en_conformer_ctc_large | test-other | 1 | 3.3 | 56.7
Table 3: Speed-up factor of 16 core Flashlight Decoder and CUDA WFST Decoder
with compact topology over single core Flashlight decoder on model-dataset
pairs studied in table 2.

###  4.1 Offline inference

For offline evaluation, we compare word error rate and throughput for several
setups. We evaluated three variants of NeMo’s Conformer CTC architecture -
Conformer CTC Small (13 million parameters), Conformer CTC Medium (30 million
parameters), and Conformer CTC Large (120 million parameters) - on Librispeech
test-clean and test-other. We evaluated in three scenarios: (1) Using the
Flashlight Decoder using 16 CPU cores, (2) using the CUDA WFST Decoder with
the “compact” CTC topology to construct $TLG$, and (3) using the CUDA WFST
Decoder with the “normal” CTC topology to construct $TLG$. We used the
Librispeech 3-gram.pruned.3e-7.arpa.gz language model in all scenarios. A
single A100-80GB PCIe card was used to run the feature extraction, acoustic
model, and - except in the Flashlight Decoder case - beam search. We ran at
batch size 200. Further hyperparameters settings are discussed below.

One area where the CUDA WFST Decoder and the Flashlight Decoder vary in terms
of hyperparameters is their concept of maintaining a list of the current most
likely hypotheses. Both have a concept of “beam width”; hypotheses with a
score gap from the best hypothesis larger than this floating point threshold
are removed. We set this value to 17.0 for both decoders. However, they have
different concepts for specifying the top number of hypotheses allowed at any
given time. The CUDA WFST Decoder allows only a maximum number of hypotheses
(called “max active”) that are defined in terms of the states of the WFST. The
Flashlight Decoder defines its maximum number of hypotheses (called “beam
size”) in terms of non-emitting CTC symbols. These are not the same because
the WFST can have states entered via epsilon arcs, and because the Flashlight
Decoder does not consider two sequences varying only in terms of blanks to be
different. We set the max active for the CUDA WFST Decoder to 10,000 while the
beam size for the Flashlight Decoder is 2,500. This choice reflects the
differences discussed above.

Surprisingly, the compact WFST consistently achieves a better word error rate
that the normal WFST. The hypothesized reason for this is that fewer tokens
are used to occupy different states that end up leading to the same output
string, since the “deduplication” paths mapping non-blank and non-epsilon
ilabels to blank olabels no longer need to be followed. The compact WFST
topology also does better than the Flashlight Decoder in terms of word error
rate, though the difference is smaller. This is likely because language model
weights are pushed left in the WFST graph and thus able to inform search
before a word is fully seen, an optimization not possible in dynamic decoders.
Reference table 1.

We document the throughputs achieved when running the end-to-end pipeline,
including feature extraction, acoustic model inference, and beam search in
table 2. According to table 2, the compact and normal topologies dominate the
Flashlight Decoder in terms of throughput, measured in terms of RTFx (hours of
audio transcribed per hour of wallclock time). Given the lower word error rate
of the compact CTC topology, we compute the throughput ratio of the CUDA WFST
Decoder with the compact topology to the Flashlight Decoder in the final
column. The throughput ratio is between 4.63 and 7.29, depending on model size
and dataset. Model size affects throughput because the acoustic model
inference time takes longer as models are larger. Slightly more time is spent
in beam search when running on test-other than on test-clean because the
acoustic model outputs less confident predictions. At a high level, this means
that fewer tokens can be pruned, which means more more data to copy in the D2H
Copy step and more work in the next iteration’s ExpandArcs.

In order to elucidate the relationship between the end-to-end pipeline
throughput increase in table 2 and the throughput increase of the beam search
decoder itself, we show the relative speed-up of the Flashlight Decoder using
16 CPU cores and the CUDA WFST Decoder vs. the Flashlight Decoder with a
single CPU core on only the beam search part of the pipeline in 3. The
Flashlight Decoder has poor scaling, achieving 4x speed-up with 16 cores
instead of closer to 16x. Even if Flashlight achieved perfect scaling, the
CUDA WFST Decoder is still several times faster.

Thus, we can conclude that the CUDA WFST Decoder, used with the compact CTC
topology, expands the Pareto frontier in terms of throughput-accuracy
tradeoffs, compared to the Flashlight Decoder.

###  4.2 Streaming inference

We evaluated latency performance in the context of a 4 layer latency-
controlled BLSTM [16]. The BLSTM had a right hand context of 20 10ms frames,
and had four BLSTM layers, with a hidden size of 600 and an internal
projection size of 300. We used the Hub5’00 corpus of phone conversations to
simulate a real-time inference scenario for real-time video conferencing
subtitling. Each request to the inference server corresponded to 600ms of
audio data, or 60 frames, each 400ms apart.

Our benchmark was implemented via a Triton [17] Python backend, taking
advantage of Triton’s ability to queue and dynamically batch data from
multiple audio streams. We sent data at a constant rate of 2.5 requests per
second (corresponding to sending 2.5 400ms windows of audio every second),
with 256 incoming audio streams per second. We benchmarked two configurations:
(1) Doing feature extraction and acoustic model inference on the GPU, but beam
search decoding on the CPU with the Flashlight Decoder; (2) Doing all parts of
inference on the GPU, using the CUDA WFST Decoder.

Results are summarized in table 4. The “Total Latency” is simply the sum of
the compute latency and the queue latency. For a full picture of latency, one
would need to add in the network latency and right hand context latency (200ms
in this case). Nevertheless, the table shows the disastrous effect of queue
latency’s quadratic dependency on compute latency. The version using the
Flashlight Decoder on CPU has a 430ms average response time, while the CUDA
WFST Decoder version achieves 54.4ms average response time, despite having
less than half the latency of the Flashlight Decoder. To make this concrete,
the average syllable duration is 100ms, so the Flashlight Decoder is over four
syllables late, while the CUDA WFST Decoder is only half a syllable late.
Furthermore, the Flashlight Decoder cannot keep up with incoming data: it
achieves RTFx of 235.4, while the maximum is 256 RTFx (because 256 audio
streams are coming in in real time). This means that the queues are constantly
growing in memory usage the longer the server runs. The only way to remedy
this is to make the server serve fewer audio streams (e.g., 128). Word error
rate between both decoders was equal. Therefore, we can conclude that for
online inference as well as offline inference, doing beam search on the GPU is
a requirement for full utilization of hardware.

Decoder | Avg Compute Latency | Avg Queue Latency | Avg Total Latency | P99 Total Latency | RTFx
---|---|---|---|---|---
CUDA Compact Topology | 30.1 | 24.4 | 54.5 | 78 | 256.0
Flashlight | 53.7 | 376.2 | 430.0 | 541.5 | 235.4
Table 4: Online Latencies and throughputs. All units are in milliseconds,
except for RTFx, which has no unit.

###  4.3 Word boosting

Fig. 6: Effect of Boost Magnitude on Word Error Rate

Figure 6 shows the word error rate of a Conformer CTC Large model with an
internal language model with a beam width of 32 evaluated on an internal 5
hour long low signal-to-noise ratio test dataset in two scenarios:

  1. 1.

The “Accurate” scenario where every unique word in the test dataset is boosted
by (the negation of) “Boost Magnitude”

  2. 2.

The “Inaccurate” scenario where only words that are not in the test set (but
are in the vocabulary of the $TLG$ model) are boosted by (the negation of)
“Boost Magnitude”. The number of out-of-vocabulary words boosted is equal to
the number of unique words in the test set.

The fact that the “Inaccurate” curve is a straight line shows that, for
reasonable boost scores, there is no disadvantage to word boosting in terms of
accuracy. Note that, in the “Accurate” scenario, word boosting helps until the
boost magnitude reaches about half the beam width, because of the phenomenon
discussed above. Therefore, as a rule of thumb, we recommend using a boost
value of at most half the beam width. Decrease in throughput and increase in
latency are miniscule; therefore, they are not depicted.

##  5 Conclusions and Future Work

To the best of our knowledge, this is the first work that investigates how to
fully utilize hardware acceleration for end-to-end automatic speech
recognition in both offline and online scenarios. We achieve between 4.6 and
7.3 times greater throughput on the same server hardware compared to the
baseline. In cloud environments where developers can dynamically provision
instances with fewer CPUs, this will reduce cost further than the 4.6 to 7.3
figure.

As far as future work is concerned we plan to extend word boosting support to
enable out-of-vocabulary words or boosting phrases using techniques described
in [18].

Last, we’d like to note, that several of the techniques described here
(dynamic batching for streaming inference, word boosting, etc.) can be applied
on top of K2’s CUDA-based RNN-T decoder [19]. This provides a path towards
high throughput, low latency inference for RNN-T models which are gaining
popularity.

## References

  * [1] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré,  “Flashattention: Fast and memory-efficient exact attention with io-awareness,” 2022.
  * [2] “Nvidia dgx a100 datasheet,” https://resources.nvidia.com/en-us-dgx-systems/dgx-ai,  Accessed: 2023-07-01.
  * [3] “Amazon ec2 p4 instances,” https://aws.amazon.com/ec2/instance-types/p4/,  Accessed: 2023-07-01.
  * [4] Hugo Braun, Justin Luitjens, and Ryan Leary,  “Gpu-accelerated viterbi exact lattice decoder for batched online and offline speech recognition,”  CoRR, vol. abs/1910.10032, 2019.
  * [5] Arturo Argueta and David Chiang,  “Decoding with finite-state transducers on GPUs,”  in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain, Apr. 2017, pp. 1044–1052, Association for Computational Linguistics.
  * [6] Zhehuai Chen, Justin Luitjens, Hainan Xu, Yiming Wang, Daniel Povey, and Sanjeev Khudanpur,  “A gpu-based WFST decoder with exact lattice generation,”  CoRR, vol. abs/1804.03243, 2018.
  * [7] “k2,” https://github.com/k2-fsa/k2,  Accessed: 2023-07-01.
  * [8] Jacob Kahn, Vineel Pratap, Tatiana Likhomanenko, Qiantong Xu, Awni Hannun, Jeff Cai, Paden Tomasello, Ann Lee, Edouard Grave, Gilad Avidov, Benoit Steiner, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert,  “Flashlight: Enabling innovation in tools for machine learning,” 2022\.
  * [9] Aleksandr Laptev, Somshubra Majumdar, and Boris Ginsburg,  “CTC variations through new WFST topologies,”  in Interspeech 2022. sep 2022, ISCA.
  * [10] Mehryar Mohri, Fernando C. N. Pereira, and Michael Riley,  “Speech recognition with weighted finite-state transducers,”  in Handbook on Speech Processing and Speech Communication, Part E: Speech recognition. 2008.
  * [11] “Getting started with cuda graphs,” https://developer.nvidia.com/blog/cuda-graphs,  Accessed: 2023-07-01.
  * [12] Robert S. Cahn,  Wide area network design: concepts and tools for optimization,  The Morgan Kaufmann series in networking. Kaufmann, San Francisco, Calif, 1998.
  * [13] “Nvidia/cucollections,” https://github.com/NVIDIA/cuCollections,  Accessed: 2023-07-01.
  * [14] “Decoding-graph creation recipe (test time),” https://kaldi-asr.org/doc/graph_recipe_test.html,  Accessed: 2023-07-01.
  * [15] C. Allauzen, M. Mohri, M. Riley, and B. Roark,  “A generalized construction of integrated speech recognition transducers,”  in 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2004, vol. 1, pp. I–761.
  * [16] Shaofei Xue and Zhijie Yan,  “Improving latency-controlled blstm acoustic models for online speech recognition,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 5340–5344.
  * [17] “Triton inference server: An optimized cloud and edge inferencing solution,” https://github.com/triton-inference-server,  Accessed: 2023-07-01.
  * [18] Cyril Allauzen and Michael Riley,  “Rapid vocabulary addition to context-dependent decoder graphs,”  in Interspeech 2015, 2015.
  * [19] Wei Kang, Liyong Guo, Fangjun Kuang, Long Lin, Mingshuang Luo, Zengwei Yao, Xiaoyu Yang, Piotr Żelasko, and Daniel Povey,  “Fast and parallel decoding for transducer,” 2022.