# Processing Natural Language on Embedded Devices: How Well Do Transformer
Models Perform?

Souvika Sarkar<EMAIL_ADDRESS>Auburn UniversityAlabamaUSA ,  Mohammad
Fakhruddin Babar<EMAIL_ADDRESS>Washington State UniversityPullmanUSA ,  Md
Mahadi Hassan<EMAIL_ADDRESS>Auburn UniversityAlabamaUSA ,  Monowar Hasan
<EMAIL_ADDRESS>Washington State UniversityPullmanUSA and  Shubhra Kanti
Karmaker Santu<EMAIL_ADDRESS>Auburn UniversityAlabamaUSA

(2024)

###### Abstract.

Voice-controlled systems are becoming ubiquitous in many IoT-specific
applications such as home/industrial automation, automotive infotainment, and
healthcare. While cloud-based voice services (e.g., Alexa, Siri) can leverage
high-performance computing servers, some use cases (e.g., robotics, automotive
infotainment) may require to execute the natural language processing (NLP)
tasks offline, often on resource-constrained embedded devices. Transformer-
based language models such as BERT and its variants are primarily developed
with compute-heavy servers in mind. Despite the great performance of BERT
models across various NLP tasks, their large size and numerous parameters pose
substantial obstacles to offline computation on embedded systems. Lighter
replacement of such language models (e.g., DistilBERT and TinyBERT) often
sacrifice accuracy, particularly for complex NLP tasks. Until now, it is still
unclear (a) whether the state-of-the-art language models, viz., BERT and its
variants are deployable on embedded systems with a limited processor, memory,
and battery power and (b) if they do, what are the “right” set of
configurations and parameters to choose for a given NLP task.

This paper presents a performance study of transformer language models under
different hardware configurations and accuracy requirements and derives
empirical observations about these resource/accuracy trade-offs. In
particular, we study how the most commonly used BERT-based language models
(viz., BERT, RoBERTa, DistilBERT, and TinyBERT) perform on embedded systems.
We tested them on four off-the-shelf embedded platforms (Raspberry Pi, Jetson,
UP2, and UDOO) with 2 GB and 4 GB memory (i.e., a total of eight hardware
configurations) and four datasets (i.e., HuRIC, GoEmotion, CoNLL, WNUT17)
running various NLP tasks. Our study finds that executing complex NLP tasks
(such as “sentiment” classification) on embedded systems is feasible even
without any GPUs (e.g., Raspberry Pi with 2 GB of RAM). Our findings can help
designers understand the deployability and performance of transformer language
models, especially those based on BERT architectures.

††journalyear: 2024††copyright: acmlicensed††conference: Proceedings of the
15th ACM/SPEC International Conference on Performance Engineering; May 7–11,
2024; London, United Kingdom††booktitle: Proceedings of the 15th ACM/SPEC
International Conference on Performance Engineering (ICPE ’24), May 7–11,
2024, London, United Kingdom††doi: 10.1145/3629526.3645054††isbn:
979-8-4007-0444-4/24/05

##  1\. Introduction

The natural language processing (NLP) domain and the emergence of large
language models rapidly transform how we interact with technology. With the
proliferation of IoT-specific applications, the demand for voice-controlled
services that can perform tasks by responding to spoken commands is growing.
Use cases of NLP applications, especially those that need embedded and mobile
systems, include home automation, healthcare, industrial control, and
automotive infotainment. To design a dialogue-based interaction system for a
target device, we need models that are feasible to (a) run on the hardware and
(b) meet the desired level of accuracy. Although some services such as digital
assistants (e.g., Alexa, Siri, Cortana) may leverage cloud resources for
processing human voices, there exist applications (e.g., offline
home/industrial robots, automotive infotainment, battlefield/military
equipment) that may not have network connectivity, thus require to synthesize
NLP tasks on the embedded device itself. The challenge is understanding the
feasibility of running a large language model on resource-limited devices.

Transformer-based architectures (Vaswani et al., 2017), especially BERT-based
models (Devlin et al., 2019), have established themselves as popular state-of-
the-art baselines for many NLP tasks, including Intent Classification (IC),
Sentiment Classification (SC), and Named Entity Recognition (NER). However, a
well-known criticism of BERT-based architectures is that they are data-hungry
and consume a lot of memory and energy; therefore, deploying them in embedded
systems is challenging. In fact, due to their excessive size (431 MB) and
parameters (110 M), deploying a pre-trained BERT model (called $BERT_{Base}$)
in resource-constrained embedded devices is often impractical, especially at
the production level with certain minimum accuracy/performance requirements.
Lighter versions of BERT (e.g., DistilBERT (Sanh et al., 2019) and TinyBERT
(Jiao et al., 2020)) often result in accuracy losses. The degree of
degradation in performance depends on the difficulty of the task, especially
since those models often cannot perform well on complex NLP tasks, including
emerging entity (Derczynski et al., 2017) or mixed emotion detection (Demszky
et al., 2020). Therefore, designers must make an inevitable trade-off between
an accurate model and one that can run smoothly in a resource-constrained
environment. Unfortunately, developers often have little idea about this
trade-off and have to spend a lot of time conducting trial-and-error
experiments to find a suitable architecture that is feasible for the target
(resource-constrained) hardware and meets a desired level of accuracy.

From a developer’s perspective, it is still unclear what is the “right” BERT-
based architecture to use for a given NLP task that can strike a suitable
trade-off between the resources available and the minimum accuracy desired by
the user. Due to the staggering size of the $BERT_{Base}$ model, we experiment
with different “distilled” BERT models (e.g., DistilBERT and TinyBERT) for IC,
SC, and NER tasks. However, existing ready-to-use distilled models perform
poorly on some SC and NER datasets (Sec. 4). Hence, there is a need to explore
other models that can better optimize the efficiency/accuracy trade-offs.

This research performs an exploratory study of BERT-based models111Note: there
exist other large language models, such as GPT (Brown et al., 2020) from
OpenAI and LaMDA (Thoppilan et al., 2022) from Google. However, they are even
more resource-hungry than BERT (thus less suitable for embedded deployment),
and some are close-sourced. Hence, our initial study limits on BERT-based
architectures. under different resource constraints and accuracy budgets to
derive empirical data about these resource/accuracy trade-offs. We aim to
answer the following questions: (a) how can we determine the suitable BERT
architecture that runs on a target hardware and meets user-defined performance
requirements (accuracy, inference time)? (b) what are the trade-offs between
accuracy and model size as we perform optimizations (such as pruning) to run
them on limited embedded device memory? and (c) what are the implications of
performing pruning on accuracy and corresponding resource usage, including
memory, inference time, and energy consumption? In answer to those questions,
we observe the overhead of running various BERT architectures on four
different hardware (viz., Raspberry Pi (Pi, 2023), Jetson Nano (Jetson, 2023),
UP2 (board, 2023), and UDOO (Bolt, 2023)). Our experiments suggest that some
BERT models (specifically those that are “distilled”) failed to achieve
desired performance goals (e.g., F1 score) for various NLP tasks. Further,
although pruning can reduce model size, it does not significantly help in
energy efficiency.

Contributions.  Our study fills the gap between simulation-based studies and
real-world scenarios, as no prior work has deployed these models on embedded
platforms. The findings of this work can help designers choose alternative
BERT-based architectures under given resource constraints, thus saving
development time and hassle. To ensure reproducibility, our implementation and
related documentation is publicly available (rep, 2024).

We made the following contributions in this paper.

  * •

Our study systematically investigates the performance of BERT-based language
models on four off-the-shelf embedded platforms (Raspberry Pi, Jetson, UP2,
and UDOO) with two different memory variants (2 GB and 4 GB RAMs). We analyzed
the trade-offs between complexity and accuracy across multiple NLP tasks.
(Sec. 3-Sec. 4).

  * •

We explore the feasibility of deploying complex NLP tasks on embedded systems
and analyze them under three metrics: (a) inference time, (b) memory usage,
and (c) energy consumption. We developed a lookup table through empirical
observations that will be useful for system designers to decide suitable model
configurations for the target platform (Sec. 4).

Our key findings.  The observations from executing the NLP tasks on our test
platforms are as follows: (a) simpler NLP tasks such as IC can result in a
relatively high (90%+) F1 score; (b) the time required to perform inference
proportional to the size of the trained model and trimming them result in some
accuracy loss (e.g., 60% of reduction in model size could reduce the accuracy
by 50%); (c) the energy consumption on our test hardware remains relatively
consistent, whether we prune the model or not; and (d) while GPUs play a role
in decreasing inference time, the unavailability of GPUs can be compensated by
faster CPUs.

We now start with the problem statement and present selected datasets (Sec.
2). Section 3 describes our experiment setup before we discuss our findings in
Sec. 4.

Figure 1. Utterance processing steps of a voice-controlled embedded device.

##  2\. Problem Statement & Datasets

We aim to study how language models can be optimally deployed to accomplish
dialog processing in embedded devices. The core technical challenge of any
dialog system is to accurately understand and interpret user “utterances” and
perform the right “action” accordingly. At a fundamental level, user utterance
understanding relies on the following three basic NLP tasks222Section 2.1
formally presents these three NLP tasks.: (a) Intent classification (IC) — to
understand the need of the user, (b) Sentiment Classification (SC) — to
understand user emotions, and (c) _Named-entity Recognition (NER)_ — to
extract related entities such as persons or objects.

Figure 1 presents the workflow of the dialogue-based systems considered in
this work. The user initiates the interaction by providing a spoken command to
the voice-controlled device (marker ① in Fig. 1). The device employs existing
automatic speech recognition techniques (Juang and Rabiner, 2005; Yu and Deng,
2016) to convert the user’s speech into texts as most language models take
textual input (②). The system then runs an “intent classifier” (Sec. 2.1.1)
and analyzes the extracted text to determine the user’s intention (③). The
classifier identifies the relevant user intentions for the given command (④).
For instance, the intents for the given command “Can you please go to my study
room and turn off the lights?” could be identified as “Motion” and “Change
operational state,” as it instructed the device to move from its current
position. Simultaneously, a “sentiment classifier”(Sec. 2.1.2) is employed to
extract the user sentiment (⑤). In this case, the extracted sentiment is
“Neutral” as the command does not express any emotion (⑥). Sentiment
classification helps the voice-controlled device grasp the emotional context
behind user utterances, enabling it to respond appropriately. In addition, the
dialog system utilizes a “named entity recognizer” (Sec. 2.1.3) to identify
specific “entities” (⑦-⑧). NER is crucial for accurately identifying user-
specified entities like locations and objects, ensuring precise execution of
commands in this scenario. For example, the entities, in this case, are “study
room” (location) and “lights” (object). Once the user’s intention and relevant
entities are identified (⑨), the control application running on the embedded
device carries out the specified task and sends a response back to the user.
In this paper, we focus on understanding how the language models perform on
embedded platforms for IC, SC, and NER tasks (e.g., steps ③-⑧).

###  2.1. NLP Tasks under Consideration

Recall from our earlier discussion that intent/sentiment classification and
named-entity recognition are fundamental NLP tasks for any voice-controlled
interactive system, chatbots, and virtual assistants. We now formally
introduce IC, SC, and NER tasks.

####  2.1.1. Intent Classification

To produce an accurate response, reduce backtracking, and minimize user
frustration, Intent Classification (IC) is needed to identify which subsequent
action a robot needs to perform depending on the user’s utterance. A formal
definition of IC can be given as follows:

###### Definition 0.

Given a collection of user utterances $U=\\{u_{1},u_{2},...,u_{n}\\}$, and a
set of intent labels $I_{x}=\\{i_{1},i_{2},...,i_{m}\\}$, classify each
utterance $u_{j}$ $\epsilon$ $U$ with one to more intents labels from $I_{x}$.

Importantly, a user might have more than one intent while speaking to a robot
and understanding the implicit or explicit intent expressed in a statement is
essential to capturing the user’s needs. For example, consider the following
command: “Can you please go to my study room and turn off the lights?” The
command wants the robot to turn off the lights in the study room; the relevant
intent here is “Change Operational State”. However, the statement also
expressed another intent related to “Motion”, as the command requires the
robot to change location. Without identifying all the underlying intents, the
system cannot perform the right next step. Hence, recognizing and
understanding all types of intents stated in an utterance is crucial for
accomplishing the eventual goal.

####  2.1.2. Sentiment Classification

Sentiment analysis is regarded as an important task for accurate user modeling
in natural dialog-based interactions, where user utterances are usually
classified into multiple emotion/sentiment labels. A formal definition can be
given as follows:

###### Definition 0.

Given a collection of user utterances $U=\\{u_{1},u_{2},...,u_{n}\\}$, and a
set of sentiment labels $S_{x}=\\{s_{1},s_{2},...,s_{m}\\}$, classify each
expression $u_{j}$ $\epsilon$ $U$ with one to more sentiment labels from
$S_{x}$.

For example, the following user utterance, “OMG, yep!!! That is the final
answer. Thank you so much!” will be classified with sentiment labels
“gratitude” and, “approval”. Similarly, statements such as “This caught me off
guard for real. I’m actually off my bed laughing” will be labeled as
“surprise” and “amusement.”

####  2.1.3. Named-entity Recognition

Named entity recognition (NER) — often referred to as entity chunking,
extraction, or identification — is a sub-task of information extraction that
seeks to locate and classify named entities mentioned in unstructured text. An
entity can be expressed by a single word or a series of words that
consistently refer to the same thing. Each detected entity is further
classified into a predetermined category. The formal definition of the NER
task can be given as follows:

###### Definition 0.

Given a collection of statements/texts $S=\\{s_{1},s_{2},...,s_{n}\\}$, and a
set of entity labels $E_{x}=\\{e_{1},e_{2},...,e_{m}\\}$, all the words/tokens
in the text will be classified with an entity label $e_{i}$ $\epsilon$
$E_{x}$.

NER can be framed as a sequence labelling task that is performed in two steps,
first, detecting the entities from the text, and second, classifying them into
different categories. A named entity recognizer model classifies each
word/phrase representing an entity into one of the four types: (a) persons
(PER), (b) objects (OBJ), (c) locations (LOC), and (d) miscellaneous names
(MISC).

###  2.2. Datasets

Our study includes the following datasets333Appendix A.1\- A.4 present
additional details about these datasets.: (a) HuRIC (for IC), (b) GoEmotion
(for SC), and (c) CoNLL and WNUT17 (for NER), as we present below.

####  2.2.1. Intent Classification: HuRIC

For IC, we use Human Robot Interaction Corpus (HuRIC) (Vanzo et al., 2020),
which is the state-of-the-art single-class classification dataset. The basic
idea of HuRIC is to build a reusable corpus for human-robot interaction in
natural language for a specific application domain, i.e., house service
robots. HuRIC includes a wide range of user utterances given to a robot
representing different situations in a house environment. The motivation
behind selecting HuRIC for our intent classification task stems from our
specific interest in utilizing a dataset that captures human-robot
conversations. The HuRIC dataset allows us to train and evaluate intent
classification models on realistic dialogues between humans and robots in
real-world scenarios. Table 1 presents some statistics of HuRIC.

Statistic | Count
---|---
Number of examples | 729
Number of intent labels | 11
Size of training dataset | 583
Size of test dataset | 146
Table 1. Statistics of HuRIC dataset.

####  2.2.2. Sentiment Classification: GoEmotion

We use GoEmotion (Demszky et al., 2020) dataset from Google AI for the SC
task. GoEmotion is a human-annotated dataset of 58,000 Reddit comments
extracted from popular English-language subreddits and labeled with 27 emotion
categories. As the largest fully annotated English language fine-grained
emotion dataset to date, the GoEmotion taxonomy was designed with both
psychology and data applicability in mind Table 2 presents some statistics of
GoEmotion. We chose the GoEmotion dataset for SC task to ensure rigorous
testing of our models. With its comprehensive emotion coverage and nuanced
labeling, GoEmotion serves as a challenging yet realistic benchmark for
evaluating sentiment detection performance.

Statistic | Count
---|---
Number of labels | 27 + Neutral
Maximum sequence length in overall datasets | 30
Size of training dataset | 43,410
Size of test dataset | 5,427
Size of validation dataset | 5,426
Table 2. Statistics of GoEmotion dataset.

####  2.2.3. Named-entity Recognition: CoNLL & WNUT17

For NER we consider two datasets, viz., CoNLL (Sang and De Meulder, 2003) and
WNUT17 (Derczynski et al., 2017).

##### CoNLL

CoNLL-2003 (Sang and De Meulder, 2003) was released as a part of CoNLL-2003
shared task: language-independent named entity recognition. The English corpus
from this shared task consists of Reuters news stories between August 1996 and
August 1997, each annotated with the entities associated with them. The data
set consists of a training file, a development file, and a test file. The
details of CoNLL-2003 are presented in Table 3.

Statistic | Articles | Sentences | Tokens
---|---|---|---
Training set | 946 | 14,987 | 203,621
Development set | 216 | 3,466 | 51,362
Test set | 231 | 3,684 | 46,435
Table 3. Statistics of CoNLL dataset.

##### WNUT17.

While the CoNLL corpus is based on news stories, we wanted to select a dataset
that contains user utterances such as those available on HuRIC. Unfortunately,
we could not find such a NER dataset but discovered a very similar corpus
(WNUT2017 (Derczynski et al., 2017)) that contains user-generated text. The
WNUT2017 dataset’s shared task focuses on identifying unusual, previously-
unseen entities in the context of emerging discussions. Identifying entities
in noisy text is really challenging, even for human annotators, due to novel
entities and surface forms. In this dataset, user comments were mined from
different social media platforms because they are large, and samples can be
mined along different dimensions, such as texts from/about geo-specific areas,
about home aid, and particular topics and events. Table 4 summarizes the
dataset properties.

Statistic | Count
---|---
Number of examples | 5690
Number of labels | 6
Size of training dataset | 3394
Size of test dataset | 1287
Size of validation dataset | 1009
Table 4. Statistics of WNUT17 dataset.

##  3\. Experimental Setup

We now summarize BERT architectures and configurations used in our experiments
(Sec. 3.1 and Sec. 3.2). We selected four off-the-shelf platforms from
different chip vendors (Intel, AMD, ARM, NVIDIA) to understand the feasibility
of using NLP tasks on a variety of architectures (Sec. 3.3). To measure the
performance of each task (IC, SC, NER), we use three popular metrics (i.e.,
Precision, Recall, and $F_{1}$ score). Section 3.4 lists the design questions
explored in our investigation. The blueprints of our implementation, including
related code/documentation, is publicly available for community use (rep,
2024).

| BERT | RoBERTa | DistilBERT | TinyBERT
---|---|---|---|---
Number of Layers | 12 | 12 | 6 | 4
Attention Heads | 12 | 12 | 12 | 12
Hidden Layer Size | 768 | 768 | 768 | 312
Feed-Forward Layer Size | 3072 | 3072 | 3072 | 1200
Vocabulary Size | 30522 | 30522 | 30522 | 30522

Table 5. Attributes of the BERT variants used in our study.

###  3.1. Off-the-Shelf BERT Variants

We use a pre-trained base variant of BERT (Devlin et al., 2018), RoBERTa (Liu
et al., 2019), DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2020)
model from Huggingface444https://huggingface.co/. and finetune the models on
respective datasets (i.e., HuRIC, GoEmotion, CoNLL, and WNUT17). Table 5
listed the parameters of the BERT variants and Table 6 presents the hyper-
parameter used in our experiments.

Hyperparameter | NER | IC/SC
---|---|---
Number of epochs | 3 | 3
Batch size | 64 | 64
Learning rate | $[e^{-6},e^{-4}]$ | $[e^{-6},e^{-4}]$
Weight decay | $[0.01,0.3]$ | $[0.01,0.3]$
Optimizer | Adam | Adam
Adam epsilon | $1e^{-8}$ | $1e^{-8}$
Max sequence length | 64 | 128
Table 6. Hyperparameter values for finetuning BERT on IC/SC and NER tasks. Embedded Platform | Architecture | CPU | GPU | Memory
---|---|---|---|---
| Raspberry Pi | ARM | Quad-core Cortex-A72 |  | 2 GB and 4 GB
| Jetson Nano | ARM | Quad-core Cortex-A57 | 128-core NVIDIA Maxwell | 2 GB and 4 GB
| UP2 | x86 | Dual-core Intel Celeron N6210 |  | 2 GB and 4 GB
| UDOO Bolt | x86 | Quad-core AMD Ryzen V1605B |  | 2 GB and 4 GB
Table 7. Embedded platforms used in our evaluations: (a)  Raspberry Pi (Pi,
2023), (b)  Jetson Nano (Jetson, 2023), (c)  UP2 (board, 2023), and (d)  UDOO
Bolt (Bolt, 2023). We used 2 GB and 4 GB memory configurations for each board.

###  3.2. Pruning and Custom Configurations

We also experiment with custom, smaller BERT configurations. Due to the
resource constraints (e.g., memory and energy limits) of embedded devices, it
is necessary to explore different variants of BERT-based models that can be
optimized to run on the device. We can reduce the model size on two fronts:
(a) by reducing the layer size and (b) by pruning various attributes.

In our study, we experiment with  _different layer combinations_ of BERT
models and test their performance on  _different hardware configurations_ ,
which are presented in Table 12, and 13. With two layers of $BERT_{Base}$
(instead of 12), the model size reduces significantly, but so does the
accuracy (in terms of $F_{1}$ score). Still, these models give better accuracy
than the distilled models on complex NLP tasks. Also, where $BERT_{Base}$
model with 12 layers cannot run on a resource-constrained device, using a
lesser number of layers enable a model to execute on tiny devices with good
accuracy compared to the distilled methods (DistilBERT and TinyBERT).

For further shrinking of the model, pruning can be applied to weights,
neurons, layers, channels, and attention heads, depending on the heuristic
used. In this paper, we focus on pruning _attention heads_ , which plays an
important role in the model’s performance and contributes a large number of
parameters. Although multi-headed attention is a driving force behind many
recent state-of-the-art models, Michel et al. (2019) finds that even if models
have been trained using multiple heads, in practice, a large percentage of
attention heads can be removed without significantly impacting performance. In
fact, in some layers, _attention heads_ can even be reduced to a single head.

Based on this fact, we experiment with reducing the size of $BERT_{Base}$ by
dynamically pruning the _attention heads_. Each attention head provides a
distribution of attention for the given input sequence. For each attention
head, we calculate the head importance by calculating the entropy of each
head. After that, we mask all the attention heads with the lowest entropy.
This process is repeated for each layer of the encoder. After masking the
heads, we calculate the overall ${F_{1}}$ score of the masked model and
determine the drop in ${F_{1}}$ score compared to the original unpruned model.
If the drop is less than a predefined threshold, we prune the masked attention
heads. We repeat the process until the drop in ${F_{1}}$ score reaches the
predefined threshold. This pruning procedure reduces the model size
significantly while maintaining the desired model performance.

###  3.3. Evaluation Platforms

We evaluate the BERT models on heterogeneous setup, viz., x86 (Intel and AMD)
and ARM platforms, including devices with or without GPU. In particular, we
used the following four different embedded platforms: (a)  Raspberry Pi 4
Model B (Pi, 2023), (b)  Jetson Nano (Jetson, 2023), (c)  UP2 (board, 2023),
and (d)  UDOO Bolt (Bolt, 2023). Table 7 lists the hardware configurations
used in our setup. Among them, only Jetson board equipped with GPU (128 NVIDIA
CUDA cores). We used 2 GB and 4 GB memory configurations for each of the four
boards. The SoC (System on Chip) of ARM-based boards (Raspberry Pi and Jetson)
configured with soldered memory — hence, we used two different boards for each
with 2 GB and 4 GB RAM configurations. The x86-based boards (i.e., UP2 and
UDOO) are modular, so we used the same board but two different DDR4 RAMs (2 GB
and 4 GB). Hence, our evaluation setup consists of eight different hardware
configurations running on four ARM and two x86 boards. For energy measurements
during the inference steps, we used UM25C energy meter (UM25C, 2017). The
total cost for our test harness is $1697 (see Table 14 in Appendix B for a
detailed breakdown). We performed all experiments on Linux kernel 5.15.0. The
NLP models were developed using PyTorch library (version 1.13).

###  3.4. Design Challenges & Research Questions

We conducted extensive experiments to investigate the following research
questions (RQs).

  * •

RQ 1. Given specific user-defined constraints, such as system resources
(processor, memory) and performance budgets (accuracy, inference time), what
is the optimal (if any) BERT-based architecture satisfying those constraints?

  * •

RQ 2. What is the accuracy vs. model-size trade-off as we prune the models?

  * •

RQ 3. What are the trade-offs of accuracy and corresponding resource usage
(e.g., memory, inference-time, energy consumptions) as we perform pruning?

  * •

RQ 4. Does GPU aid in inference time?

  * •

RQ 5. What are the energy consumption differences among various architectures
(x86 and ARM)? Does the presence of GPUs impact energy usage?

##  4\. Results

In this section, we report our results for the three basic NLP tasks, i.e.,
IC, SC, and NER for both existing (e.g., BERT, RoBERTa, DistilBERT, and
TinyBERT) and custom BERT architectures.

Intent Classification Task (Dataset: HuRIC)
---
BERT | RoBERTa | DistilBERT | TinyBERT
Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$
0.943 | 0.985 | 0.961 | 0.975 | 0.952 | 0.962 | 0.951 | 0.903 | 0.927 | 0.912 | 0.897 | 0.902

Table 8.  Performance of BERT, RoBERTa, DistilBERT, and TinyBERT for the IC
task.

Multi-label Sentiment Detection Task (Dataset: GoEmotion)
---
BERT | RoBERTa | DistilBERT | TinyBERT
Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$
0.77 | 0.37 | 0.490 | 0.731 | 0.452 | 0.567 | 0.174 | 0.121 | 0.134 | 0.060 | 0.030 | 0.031

Table 9.  Performance of BERT, RoBERTa, DistilBERT, and TinyBERT for the SC
task.

Named-entity Recognition Task
---
Dataset: CoNLL
BERT | RoBERTa | DistilBERT | TinyBERT
Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$
0.891 | 0.963 | 0.926 | 0.882 | 0.955 | 0.917 | 0.906 | 0.967 | 0.935 | 0.872 | 0.958 | 0.911
Dataset: WNUT17
BERT | RoBERTa | DistilBERT | TinyBERT
Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$ | Precision | Recall | $\boldsymbol{F_{1}}$
0.671 | 0.295 | 0.410 | 0.537 | 0.315 | 0.397 | 0.316 | 0.014 | 0.028 | 0.0 | 0.0 | 0.0

Table 10.  Performance of BERT, RoBERTa, DistilBERT, and TinyBERT for the NER
task.

###  4.1. Experience with Existing BERT Variants

##### Intent Classification (IC)

Recall from our earlier discussion (Sec. 2.2.1) that we used the HuRIC dataset
for IC. Table 8 presents our findings after running IC tasks using the HuRIC
dataset on different hardware. We observed that all the models performed
similarly on this dataset, achieving more than 90% $F_{1}$ Score.

##### Multi-label Sentiment Classification (SC)

We next analyze the performance of multi-label SC tasks on Raspberry Pi,
Jetson, UP2, and UDOO board. As mentioned in Sec. 2.2.2, we used GoEmotion
(Demszky et al., 2020) dataset for this task. GoEmotion includes direct user-
to-user conversation text and labels them with many user emotions. Table 9
summarizes the performance of all the models for this task. Interestingly, for
this task, DistilBERT and TinyBERT failed drastically, as they achieved a very
low $F_{1}$ Score. The failure of distilled models can be attributed to the
difficulty of the task. Multi-label SC requires each utterance to be
classified with more than one sentiment. Therefore, this dataset is not a
straightforward positive-negative sentiment detection.

##### Named-entity Recognition (NER)

As we mention in Sec. 2.2.3, we use two different datasets to test the NER
task. Table 10 summarizes the performance over both the NER datasets and shows
that for the CoNLL dataset. In this setup, all models performed comparatively
the same. However, the performance of distilled models dropped sharply for the
WNUT17 dataset (which focuses on identifying unusual, previously-unseen
entities). This drop tends to be due to the difficulty of analyzing this task,
as NER evaluates the ability to detect and classify novel, emerging,
singleton-named entities in noisy inputs.

In summary, our findings are as follows.

• All models achieved decent F1 scores (¿90%) for IC task. • DistilBERT and
TinyBERT struggled with the multi-label SC task as none achieved an F1 score
of more than 15%. • All BERT models excelled in the NER task on the CoNLL
dataset and accurately recognized named entities (resulting in ¿90% F1
scores). • Distilled models showed a performance decline for the NER task on
the WNUT17 dataset with the F1 score dropping to less than 5%, indicating
difficulty with dataset intricacies.

Task Metrics $\boldsymbol{F}_{1}$ Score Threshold ($\boldsymbol{\theta}$)
$\boldsymbol{\theta}_{50}$ $\boldsymbol{\theta}_{60}$
$\boldsymbol{\theta}_{70}$ $\theta_{80}$ $\theta_{90}$ IC Layer MS Params
Pruning 2 4 6 8 144.3 N/A N/A N/A 35.1 68% 2 4 6 8 148.3 N/A N/A N/A 37.1 66%
2 4 6 8 N/A 195.6 N/A 282.2 48.9 70.5 56% 36% 2 4 6 8 154.2 198.7 246 N/A 38.6
49.7 61.5 65% 55% 44% 2 4 6 8 N/A 211.3 268 303.5 52.8 67 75.9 52% 39% 31% NER
Layer MS Params Pruning 2 4 6 8 136.4 N/A N/A N/A 34.1 67% 2 4 6 8 147.4 N/A
N/A N/A 36.9 65% 2 4 6 8 N/A 185.2 233.3 N/A 46.3 58.3 57% 45% 2 4 6 8 N/A 201
249.8 268 46.3 62.4 67 45% 57% 38% 2 4 6 8 N/A N/A 260.8 289.2 65.2 72.3 39%
32%

Table 11.  Performance of SC and NER tasks for the GoEmotion and WNUT17
datasets on various configurations. In metrics column, MS= Model Size (MB),
Params= Parameters (Million).

###  4.2. Exploration with Custom Architectures

Based on our experiment results (Tables 8–10), we further explore different
alternative BERT-based architectures by reducing the layers and pruning the
attention heads from the original $BERT_{Base}$ model. For this exploration,
we primarily focus on the challenging tasks, i.e., multi-label SC and NER
where off-the-shelf models (e.g., DistilBERT and TinyBERT) failed to perform.

Table 11–Table 13 present our exploration findings.555Note: We omit the
results for IC on custom BERT architectures as existing models suffice to
perform this task. We discuss our observations in Sec. 4.2.3 and provide
answers to the research questions posed in Sec. 3.4. Before we proceed with
the discussion, we present a brief overview of the attributes and metrics used
in our evaluation.

####  4.2.1. Model Attributes

In the evaluation, we vary the following model attributes.

  * •

${F_{1}}$ Threshold ($\theta$): The $\theta$-cells represents what percentage
of the ${F_{1}}$ score (with respect to $BERT_{Base}$) is retained by the
models. In our experiment, we varied $\theta$ between 50% to 90% and reported
the model details in respective columns. For example, $\theta$ set to 80%
implies the $\theta_{80}$ column.

  * •

Platform: Indicates the memory capacity of the different hardware we used in
our exploration.

  * •

Layer: Represents the number of layers retained.

  * •

Model Size: The size of the saved model after training. We stored the saved
model on the disk which is then loaded on the memory for inference.

  * •

Parameters: This metric indicates the total number of parameters in the saved
model. We obtained the model parameters using the model.parameters() method in
PyTorch (Paszke et al., 2017).

  * •

Pruning: Pruning percentage represents the reduction in size from the
$BERT_{Base}$ model. For example, a pruning percentage of 70% implies that the
pruned model is 70% smaller than $BERT_{Base}$.

####  4.2.2. Performance Metrics

We consider the three metrics to benchmark the NLP models: (a) inference time,
(b) memory usage, and (c) energy consumption, as we present below.

  * •

Memory Consumption: Maximum memory usage (in megabytes) of the corresponding
NLP task running on different boards during the inference time. We used Python
memory_profiler for each input to get the memory usage.

  * •

Inference Time: Depending on the specific task, the 95th percentile time
required for the model to infer the appropriate Intent, Sentiment or Entity
from a given command.

  * •

Energy Consumption: The average energy consumed (in watts) by the different
hardware during the inference of a given command. We measured both the rest
time (i.e., when the system is idle) and inference time energy consumption.

Plat. Metrics $\boldsymbol{F}_{1}$ Score Threshold ($\boldsymbol{\theta}$)
$\boldsymbol{\theta}_{50}$ $\boldsymbol{\theta}_{60}$
$\boldsymbol{\theta}_{70}$ $\theta_{80}$ $\theta_{90}$ Pi 2 GB Layer EC MC IT
2 4 6 8 4.24 N/A N/A N/A 394.3 0.52 2 4 6 8 4.28 N/A N/A N/A 391.5 0.51 2 4 6
8 N/A 4.32 N/A 4.37 440.4 563 1.1 2.17 2 4 6 8 4.35 4.3 4.48 N/A 401.9 446.2
497.3 0.54 1.01 1.69 2 4 6 8 N/A 3.91 4.45 4.34 462.9 521.5 559.4 1.3 1.81
2.35 Pi 4 GB Layer EC MC IT 2 4 6 8 4.91 N/A N/A N/A 397.5 0.44 2 4 6 8 5.05
N/A N/A N/A 401 0.44 2 4 6 8 N/A 4.67 N/A 5.09 446.3 569.2 0.82 1.5 2 4 6 8
4.98 4.72 4.8 N/A 404.2 453.2 507.7 0.5 0.81 1.21 N/A 4 6 8 4.77 4.78 4.95
464.6 528.8 563.4 1.00 1.46 1.55 Jetson 2 GB Layer EC MC IT 2 4 6 8 5.87 N/A
N/A N/A 299.2 0.29 2 4 6 8 5.96 N/A N/A N/A 320.1 0.26 2 4 6 8 N/A 6.01 N/A
6.28 345.4 478.5 0.49 0.88 2 4 6 8 5.87 6.14 6.26 N/A 329.7 352.6 415 0.30
0.51 0.68 2 4 6 8 N/A 6.13 6.22 6.24 367.5 429.8 463.9 0.56 .797 1.02 Jetson 4
GB Layer EC MC IT 2 4 6 8 6.27 N/A N/A N/A 355.5 0.27 2 4 6 8 6.25 N/A N/A N/A
359.7 0.27 2 4 6 8 N/A 6.21 N/A 6.43 410.1 532.9 0.45 0.86 2 4 6 8 6.27 6.31
6.36 N/A 366 408.8 459.4 0.30 0.46 0.66 2 4 6 8 N/A 6.39 6.4 6.47 423 488.4
524 0.50 0.76 0.87 UP2 2 GB Layer EC MC IT 2 4 6 8 10.897 N/A N/A N/A 670.9
0.12 2 4 6 8 10.9 N/A N/A N/A 708.6 0.12 2 4 6 8 N/A 10.94 N/A 10.79 606.8
741.7 0.21 0.39 2 4 6 8 10.96 11.05 10.78 N/A 696.8 693 742.6 0.13 0.21 0.30 2
4 6 8 N/A 10.81 10.73 10.76 741.1 707.4 782.1 0.23 0.31 0.42 UP2 4 GB Layer EC
MC IT 2 4 6 8 10.17 N/A N/A N/A 738.3 0.12 2 4 6 8 11.05 N/A N/A N/A 653.2
0.12 2 4 6 8 N/A 10.04 N/A 11.51 715.7 870 0.21 0.40 2 4 6 8 10.99 11.20 11.38
N/A 736.8 753.4 792.9 0.13 0.22 0.30 2 4 6 8 N/A 11.01 11.38 11.16 736.2 798.3
875.3 0.24 0.30 0.41 UDOO 2 GB Layer EC MC IT 2 4 6 8 23.08 N/A N/A N/A 379.3
0.06 2 4 6 8 23.12 N/A N/A N/A 424.6 0.06 2 4 6 8 N/A 23.43 N/A 23.64 361.5
422 0.10 0.12 2 4 6 8 23.28 23.38 23.62 N/A 377.9 348.1 461 0.06 0.09 0.13 2 4
6 8 N/A 23.57 23.56 23.46 360.9 468.2 453.1 0.10 0.14 0.17 UDOO 4 GB Layer EC
MC IT 2 4 6 8 23.076 N/A N/A N/A 450.1 0.07 2 4 6 8 23.12 N/A N/A N/A 445.4
0.07 2 4 6 8 N/A 22.78 N/A 23.18 487.3 596.8 0.11 0.16 2 4 6 8 22.98 23.05
22.99 N/A 443.8 505.9 547.8 0.08 0.11 0.15 2 4 6 8 N/A 23.13 22.96 23.15 505.7
544.7 601.9 0.12 0.16 0.16

Table 12.  Performance of SC task for the GoEmotion dataset on various
configurations. In metrics column, EC=Energy Consumption (W), MC= Memory (MB),
and IT= Inference Time (s). The gray cells highlight the best-case scenario
for each $F_{1}$ threshold ($\theta$).

Plat. Metrics $\boldsymbol{F}_{1}$ Score Threshold ($\boldsymbol{\theta}$)
$\boldsymbol{\theta}_{50}$ $\boldsymbol{\theta}_{60}$
$\boldsymbol{\theta}_{70}$ $\theta_{80}$ $\theta_{90}$ Pi 2 GB Layer EC MC IT
2 4 6 8 4.28 N/A N/A N/A 676.3 0.55 2 4 6 8 4.36 N/A N/A N/A 709 0.55 2 4 6 8
N/A 4.3 4.51 N/A 721.3 678.4 1.08 0.61 2 4 6 8 N/A 3.95 4.35 4.51 736.8 700.3
678.4 1.19 0.61 0.61 2 4 6 8 N/A N/A 4.38 4.53 704.7 692.2 0.56 0.65 Pi 4 GB
Layer EC MC IT 2 4 6 8 4.48 N/A N/A N/A 675 0.49 2 4 6 8 4.53 N/A N/A N/A
698.3 0.516 2 4 6 8 N/A 4.62 4.63 N/A 683.6 698.9 0.57 0.57 2 4 6 8 N/A 4.59
4.85 4.82 699.1 706 686 0.55 0.57 0.63 2 4 6 8 N/A N/A 4.9 4.88 706.6 709 0.60
0.65 Jetson 2 GB Layer EC MC IT 2 4 6 8 5.74 N/A N/A N/A 314.6 0.29 2 4 6 8
5.8 N/A N/A N/A 348.2 0.29 2 4 6 8 N/A 5.96 5.7 N/A 388.7 357.7 0.45 0.29 2 4
6 8 N/A 6.05 5.77 5.7 392.4 359 362.7 0.50 0.31 0.29 2 4 6 8 N/A N/A 5.8 5.71
362.3 368.9 0.33 0.29 Jetson 4 GB Layer EC MC IT 2 4 6 8 5.74 N/A N/A N/A
368.3 0.29 2 4 6 8 5.76 N/A N/A N/A 365.8 0.27 2 4 6 8 N/A 6.23 5.95 N/A 418.5
368.7 0.497 0.29 2 4 6 8 N/A 6.04 6.12 6.04 424.4 365.5 361.7 0.45 0.33 0.29 2
4 6 8 N/A N/A 6.05 6.08 363.5 366.6 0.32 0.30 UP2 2 GB Layer EC MC IT 2 4 6 8
11.108 N/A N/A N/A 512.5 0.13 2 4 6 8 11.2 N/A N/A N/A 597.6 0.12 2 4 6 8 N/A
11.18 11.44 N/A 652.9 574 0.19 0.18 2 4 6 8 N/A 11.21 10.77 10.72 650.8 601.8
603.5 0.20 0.12 0.12 2 4 6 8 N/A N/A 10.63 10.7 591.3 591.1 0.12 0.13 UP2 4 GB
Layer EC MC IT 2 4 6 8 10.99 N/A N/A N/A 599.8 0.12 2 4 6 8 11.21 N/A N/A N/A
601.8 0.11 2 4 6 8 N/A 11.36 11.33 N/A 651.4 658.1 0.19 0.20 2 4 6 8 N/A 11.32
11.40 11.35 653.7 652.7 657.7 0.19 0.19 0.22 2 4 6 8 N/A N/A 11.16 11.3 657.7
652.9 0.22 0.21 UDOO 2 GB Layer EC MC IT 2 4 6 8 24.02 N/A N/A N/A 437.8 0.07
2 4 6 8 24.08 N/A N/A N/A 438.6 0.05 2 4 6 8 N/A 24.09 24.07 N/A 491.2 490.8
0.09 0.09 2 4 6 8 N/A 24.14 24.06 24.02 490.9 490.5 503.4 0.09 0.09 0.09 2 4 6
8 N/A N/A 24.11 24.07 490.3 492.4 0.09 0.08 UDOO 4 GB Layer EC MC IT 2 4 6 8
22.79 445.6 0.06 2 4 6 8 22.86 443 0.05 2 4 6 8 23.07 23.57 500.1 496.4 0.08
0.09 2 4 6 8 23.09 23.49 23.39 497.5 499.5 510.1 0.09 0.08 0.09 2 4 6 8 23.58
23.50 495.2 499.8 0.08 0.08

Table 13.  Performance of NER task for the WNUT17 dataset on various
configurations. In metrics column, MS= Model Size (MB), Params= Parameters
(Million), EC=Energy Consumption (Watt), MC= Memory (MB), and IT= Inference
Time (s). The gray cells highlight the best-case scenario for each $F_{1}$
threshold ($\theta$).

####  4.2.3. Observations

We now discuss our major observations and address the research questions
introduced in Sec. 3.4.

##### Selecting “suitable” model subject to given constraints [RQ 1].

We can address this specific research question by inspecting Table 12 and 13.
Note that Table 12 and 13 provide information on the model size, performance,
parameters, and pruning for the SC and NER tasks, respectively. Let us assume
a system designer is looking for suitable NER models for a 2 GB embedded
platform that maintains approximately 70% of BERT’s accuracy ($\theta_{70}$).
In this case, we can (a) scan through the NER performance metrics (i.e., Table
13), and (b) observe from Pi 2 GB Platform row and $\theta_{70}$ column that a
six-layered and pruned (45% reduced) BERT model can run on a Pi 2 GB platform
and attain 70% of $BERT_{Base}$’s original $F_{1}$ score. Hence, our
exploration (and similar experiments along this line) can aid the designers to
select appropriate models with desired performance guarantees.

##### Accuracy and model-size trade-offs for pruned architectures [RQ 2].

Table 12 and Table 13 further provide insights on the pruning vs. $F_{1}$
score trade-off. For example, in Table 12, the Pi 2 GB Platform row shows a
set of models that can run on that system. The same row and $\theta_{80}$ (80%
$F_{1}$ score threshold) column show that even pruning 55% of a $BERT_{Base}$
model with four layers can retain 80% of original $F_{1}$ score of
$BERT_{Base}$, while the model size can be reduced to 198.7 MB from 441.55 MB.
Tables 12 and 13 indicate that although pruning has only a minor impact on
memory consumption, it does not have a significant effect on energy
consumption. Furthermore, our analysis of Table 12 suggests that inference
time is directly proportional to the size of the model, implying that
decreasing the model size leads to a decrease in inference time.

##### Accuracy vs. system resource trade-offs for pruned architectures [RQ 3].

If a user has precise requirements for inference time and memory consumption
for a given hardware, one can scan Table 12 and 13 to pick the optimum model
that meets those requirements. For instance, if we want to find NER models
that can make inferences in less than 0.56 seconds on Raspberry Pi that has 4
GB of memory, the corresponding Platform row Pi (4 GB) in Table 13, shows us
the model parameters that can satisfy this requirement(e.g., two-layered,
four-layered). Since both of them are feasible for the chosen platform,
designers can choose any of them based on the required application
performance. As an example, if we pick a two-layered BERT model, the accuracy
is 60%, and the memory consumption is 698.3 MB. In contrast, if we select the
four-layered BERT model, it can achieve 80% accuracy, with a cost of higher
memory consumption of 699.1 MB. Hence, at the expense of slightly higher
memory consumption, it is possible to get 20% more accuracy. Such a lookup-
based approach allows the designers to perform a desired cost-benefit
analysis.

##### The case for GPUs [RQ 4].

Intuitively, GPUs aid in any learning-enabled tasks. We used one GPU-enabled
hardware (Jetson Nano) in our design space exploration. As Table 12 and Table
13 illustrate Jetson reduces inference time compared to the Raspberry Pi board
(no GPU). However, GPUs alone cannot provide faster inference. For instance,
x86 boards (UP2 and UDOO) do not have GPUs but are equipped with a faster
processor, and hence result in better inference times (i.e., took less time to
process the queries). Systems with better hardware (CPU/GPU/memory) can output
inference decisions faster, which may increase power consumption, as we
discuss next.

Figure 2. Energy consumption during rest mode and inference period. ARM
devices use less energy compared to x86 systems.

##### Energy consumption on various architectures and model configurations [RQ
5].

Since many embedded platforms used for NLP tasks (e.g., voice-controlled
robots, voice assistants, IoT devices) are battery-operated, energy
consumption for inferring user commands is a crucial parameter. Hence, we also
analyze the energy usage of the NLP tasks. For any selected BERT model, one
can find the system energy consumption from Table 12 and 13, for two different
tasks, respectively. As the table shows, (for a given hardware) during the
inference of a given command, energy consumption does not vary significantly
for various models.

To understand the energy usage of the various NLP tasks on our test platforms,
we also measured the energy consumption of each board during rest mode and
inference period (see Fig. 2). We obtained the rest mode energy usage by
idling the device for 10 minutes and took the average value. For inference
energy consumption, we tested with 40 SC and 40 NER queries and repeated each
of them 100 times (i.e., a total of $2\times 40\times 100=8000$ samples. We
report the maximum (lighter gray) bar and minimum (darker gray) energy
consumption values of the 8000 trials.

As Fig. 2 shows, the inference energy consumption increases by a factor of
1.73 to 4.89 times compared to the rest mode energy usage. Besides, ARM
architectures (Raspberry Pi and Jetson) consume less energy than the x86
architectures (UP2 and UDOO). Another interesting observation is that even
though Jetson boards use GPU, they are more power efficient for performing NLP
tasks compared to some CPU-only x86 systems. Our experiments show that the AMD
Ryzen platform (UDOO) performs poorly in terms of energy usage. However, as
Table 12 and Table 13 indicate, UDOO boards output faster inference time
(since they have relatively faster CPU than the others). Hence, there exists a
trade-off between inference time and energy usage.

####  4.2.4. Summary of Findings

Our key findings for custom BERT architectures are listed below.

Model Size & Pruning. • Pruning helps in reduction in size (upto 67%) while
maintaining at least 50% of $BERT_{Base}$’s F1 score. • The time required for
inference is directly related to the size of the model, i.e., a smaller model
size results in a reduction in inference time. • Pruning of attention heads
does not reduce memory usage. • Pruning attention heads does not improve
energy consumption significantly.

System Artifacts. • Faster x86 (Intel and AMD) platforms outperform ARM SoCs
(e.g., Pi and Jetson boards) wrt. inference time, but their energy consumption
is significantly higher (e.g., at least 2.60 times) than ARM counterparts. •
GPUs aid in performance (e.g., inference times are approx. 2-times faster in
Jetson than Raspberry Pi) but GPUs alone in Jetson boards cannot outperform a
relatively faster CPU (i.e., those used in UP2 and UDOO boards). • Powerful
processor can decrease inference time (as expected) but comes with a cost
(increased power consumption: 2.60-5.90 times higher).

##  5\. Discussion

We explore different custom architectures of BERT-based language models and
test their deployment feasibility in low-power embedded devices. We conducted
extensive performance evaluations on four embedded platforms from various
vendors with varying computing capabilities to cover a wide range of
application scenarios. We show that it is not always feasible to shrink the
size of “finetuned” $BERT_{Base}$ model that can satisfy specific user-defined
accuracy/performance budgets. We also report which models are deployable to
resource-constrained devices with given user requirements. We believe our
empirical findings will help the developers quickly narrow down the plausible
BERT-based architecture for target applications, thus saving development time
and effort. While we tested the NLP models on four embedded platforms (a total
of 8 hardware configurations) in a Linux environment, they can be ported to
other systems, such as smartphones/tablets running different OSes (such as
Android). Thus our empirical study is applicable in broader human-centric
application domains, including chatbots (Adamopoulou and Moussiades, 2020;
Brandtzæg and Følstad, 2017; Shum et al., 2018), virtual assistants (White,
2018; Schmidt et al., 2018), and language translation (Oettinger, 2013; Brown
et al., 1988).

Our study is limited to BERT-based models for four existing datasets (i.e.,
may not generalize to other language models and datasets). However, our
evaluation framework is modular and can be retrofitted to other
architectures/datasets without loss of generality. While shrinking models have
made it possible to deploy them on resource-constrained embedded devices,
their performance on new datasets or tasks is often limited. One potential
solution to mitigate this issue is to utilize continual learning techniques
(De Lange et al., 2021; Hadsell et al., 2020), as they allow models to
continuously learn and evolve based on increasing data input while retaining
previously acquired knowledge. Our future work will explore the feasibility of
employing continual learning for embedded devices.

One of the challenges to figuring out the optimal BERT-based architecture is
the lack of application-specific (viz., voice-controlled robots for home
automation) datasets. Existing datasets either (a) do not have enough examples
for training deep learning models or (b) do not provide complex, practical
queries to test the robustness of a given model. Building suitable datasets
for IoT-specific human-centric applications such as voice-control
home/industrial automation is an interesting open research problem.

##  6\. Related Work

We discuss related research on two fronts: (a) BERT-based models and their
efficient variants and (b) using NLP on embedded devices.

###  6.1. BERT-based Models and their Variants

The performance of BERT comes at a high computation and memory cost, which
makes on-device inference really challenging. To mitigate this issue,
researchers have proposed knowledge distillation approaches from the original
BERT model, for example, (a) “finetune” the BERT model to improve task-
specific knowledge distillation (Turc et al., 2019; Tsai et al., 2019), (b)
use Bi-LSTM models (Tang et al., 2019) for knowledge distillation from BERT,
(c) leverage single-task models to teach a multi-task model (Clark et al.,
2019), (d) distillation of knowledge from an ensemble of BERT into a single
BERT (Liu et al., 2020), (e) TinyBERT (Jiao et al., 2020) uses a layer-wise
distillation strategy for BERT in both the pre-training and fine-tuning
stages, and (f) DistilBERT (Sanh et al., 2019) halves the depth of the BERT
model through knowledge distillation in the pre-training stage and an optional
fine-tuning stage. On a different direction, the Patient Knowledge
Distillation approach (Sun et al., 2019) compresses an original large model
(“teacher”) into an equally-effective lightweight shallow network (“student”).
Other BERT models (e.g., SqueezeBERT (Iandola et al., 2020), MobileBERT (Sun
et al., 2020), Q8BERT (Zafrir et al., 2019), ALBERT (Lan et al., 2020)) can
also reduce resource consumption than the vanilla BERT. EdgeBERT (Tambe et
al., 2021), an algorithm-hardware co-design approach, performs latency-aware
energy optimizations for multi-task NLP problems. However, unlike ours,
EdgeBERT (a) does not apply attention heads pruning, and (b) does not report
scores on downstream NLP tasks on real-world embedded systems.

###  6.2. NLP for Embedded Platforms

Researchers have explored NLP techniques to facilitate natural communication
between humans and embedded devices, especially in the context of voice-
controlled cognitive robots. For example, Megalingam et al. (2019) presents a
voice recognition tool that compares the user’s input commands with the stored
data. Zhang et al. (2018) propose a ROS-based robot that analyzes commands
using an offline grammar recognition library. Megalingam et al. (2021) propose
a cost-efficient speech processing module running on ROS that can provide
natural language responses to the user. There also exists ROS-integrated
independent speech recognition packages (Zhang and Xu, 2015; Sharan et al.,
2019) as well as Arduino-based (Verma et al., 2020) and custom (Lv et al.,
2008) voice-control robot platforms. House et al. (2009) a voice-controlled
robotic arm (named VoiceBot) for individuals with motor impairments (Bilmes et
al., 2005). However, most of these works focused on rule-based approaches, and
we note that transformer architectures are still under-explored in terms of
their practical deployment challenges in real-world embedded and robotic
devices, which is the focus of this study.

###  6.3. Uniqueness of Our Work

While existing work can reduce the size of BERT models through distillation
and pruning, from a system design perspective, it is still difficult and
tedious for a developer to find out the “right” BERT-based architecture to use
in an embedded platform. To date, it is also unclear which lighter version of
BERT would find the optimal balance between the resources available in an
embedded device (e.g., CPU, GPU, memory) and the minimum accuracy desired. We
used four off-the-shelf platforms widely used by developers for various IoT
and embedded applications and benchmarked state-of-the-art BERT architectures.
Our empirical evaluation and design space exploration on heterogeneous
platforms (e.g., x86 and ARM, with or without GPU) can help the system and
machine learning engineers to pick suitable architectures depending on target
system configuration and performance constraints (e.g., accuracy, $F_{1}$
score). To the best of our knowledge, this work is one of the first efforts to
study the feasibility of deploying BERT-based models in real-world resource-
constrained embedded platforms.

##  7\. Conclusion

This paper presents an empirical study of BERT-based neural architectures in
terms of the feasibility of deploying them on resource-constrained systems,
which have become ubiquitous nowadays. Our performance evaluation results will
assist developers of multiple ubiquitous computing domains, such as voice-
controlled home and industrial automation, precision agriculture, and medical
robots to determine the deployability of NLP models in their target platform.
By using our benchmark data, designers of ubiquitous systems will now be able
to select the “right” hardware, architecture, and parameters depending on the
resource constraints and performance requirements. This will also save time on
the developer’s end, as they can make informed choices regarding which BERT-
based architecture to use during development based on their NLP application
scenario and the available hardware.

###### Acknowledgements.

This work has been partially supported by the National Science Foundation
Standard Grant Award 2302974, Air Force Office of Scientific Research
Grant/Cooperative Agreement Award FA9550-23-1-0426, and Washington State
University Grant PG00021441. Any findings, opinions, recommendations, or
conclusions expressed in the paper are those of the authors and do not
necessarily reflect the views of sponsors.

## References

  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , J. Burstein, C. Doran, and T. Solorio, Eds. Association for Computational Linguistics, 2019, pp. 4171–4186. [Online]. Available: https://doi.org/10.18653/v1/n19-1423
  * Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter,” _CoRR_ , vol. abs/1910.01108, 2019. [Online]. Available: http://arxiv.org/abs/1910.01108
  * Jiao et al. (2020) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “Tinybert: Distilling BERT for natural language understanding,” in _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , ser. Findings of ACL, T. Cohn, Y. He, and Y. Liu, Eds., vol. EMNLP 2020. Association for Computational Linguistics, 2020, pp. 4163–4174. [Online]. Available: https://doi.org/10.18653/v1/2020.findings-emnlp.372
  * Derczynski et al. (2017) L. Derczynski, E. Nichols, M. van Erp, and N. Limsopatham, “Results of the WNUT2017 shared task on novel and emerging entity recognition,” in _Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017_ , L. Derczynski, W. Xu, A. Ritter, and T. Baldwin, Eds. Association for Computational Linguistics, 2017, pp. 140–147. [Online]. Available: https://doi.org/10.18653/v1/w17-4418
  * Demszky et al. (2020) D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “Goemotions: A dataset of fine-grained emotions,” _arXiv preprint arXiv:2005.00547_ , 2020.
  * Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” _CoRR_ , vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
  * Thoppilan et al. (2022) R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C. Chang, I. Krivokon, W. Rusch, M. Pickett, K. S. Meier-Hellstern, M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. H. Chi, and Q. Le, “Lamda: Language models for dialog applications,” _CoRR_ , vol. abs/2201.08239, 2022. [Online]. Available: https://arxiv.org/abs/2201.08239
  * Pi (2023) R. Pi, “https://www.raspberrypi.com/,” 2023, accessed:May 15. [Online]. Available: https://www.raspberrypi.com/
  * Jetson (2023) Jetson, 2023, accessed:Sep 15. [Online]. Available: https://developer.nvidia.com/embedded/jetson-nano-developer-kit
  * board (2023) U. board, 2023, accessed:Sep 15. [Online]. Available: https://up-board.org/
  * Bolt (2023) U. Bolt, 2023, accessed:Sep 15. [Online]. Available: https://www.udoo.org/discover-the-udoo-bolt/
  * rep (2024) “Transformers bechmarking: Implementation and artifacts,” 2024. [Online]. Available: https://github.com/CPS2RL/Benchmarking-BERT-Embedded-Devices-ICPE24-Artifact
  * Juang and Rabiner (2005) B.-H. Juang and L. R. Rabiner, “Automatic speech recognition–a brief history of the technology development,” _Georgia Institute of Technology. Atlanta Rutgers University and the University of California. Santa Barbara_ , vol. 1, p. 67, 2005.
  * Yu and Deng (2016) D. Yu and L. Deng, _Automatic speech recognition_. Springer, 2016, vol. 1.
  * Vanzo et al. (2020) A. Vanzo, D. Croce, E. Bastianelli, R. Basili, and D. Nardi, “Grounded language interpretation of robotic commands through structured learning,” _Artificial Intelligence_ , vol. 278, 2020. [Online]. Available: https://doi.org/10.1016/j.artint.2019.103181
  * Sang and De Meulder (2003) E. F. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” _arXiv preprint cs/0306050_ , 2003.
  * Devlin et al. (2018) J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” _CoRR_ , vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
  * Michel et al. (2019) P. Michel, O. Levy, and G. Neubig, “Are sixteen heads really better than one?” in _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 14 014–14 024. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html
  * UM25C (2017) UM25C, 2017. [Online]. Available: https://www.makerhawk.com/collections/frontpage/products/makerhawk-um25c-usb-tester-bluetooth-usb-meter-type-c-current-meter-usb-power-meter-dc-24-000v-5-0000a-usb-cable-tester-1-44-inch-color-lcd-multimeter-voltage-tester-usb-load-qc-2-0-qc-3-0
  * Paszke et al. (2017) A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * Adamopoulou and Moussiades (2020) E. Adamopoulou and L. Moussiades, “Chatbots: History, technology, and applications,” _Machine Learning with Applications_ , vol. 2, p. 100006, 2020\.
  * Brandtzæg and Følstad (2017) P. B. Brandtzæg and A. Følstad, “Why people use chatbots,” in _Internet Science - 4th International Conference, INSCI 2017, Thessaloniki, Greece, November 22-24, 2017, Proceedings_ , ser. Lecture Notes in Computer Science, I. Kompatsiaris, J. Cave, A. Satsiou, G. Carle, A. Passani, E. Kontopoulos, S. Diplaris, and D. McMillan, Eds., vol. 10673\. Springer, 2017, pp. 377–392. [Online]. Available: https://doi.org/10.1007/978-3-319-70284-1_30
  * Shum et al. (2018) H.-Y. Shum, X.-d. He, and D. Li, “From eliza to xiaoice: challenges and opportunities with social chatbots,” _Frontiers of Information Technology & Electronic Engineering_, vol. 19, pp. 10–26, 2018.
  * White (2018) R. W. White, “Skill discovery in virtual assistants,” _Communications of the ACM_ , vol. 61, no. 11, pp. 106–113, 2018.
  * Schmidt et al. (2018) B. Schmidt, R. Borrison, A. Cohen, M. Dix, M. Gärtler, M. Hollender, B. Klöpper, S. Maczey, and S. Siddharthan, “Industrial virtual assistants: Challenges and opportunities,” in _Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers_ , 2018, pp. 794–801.
  * Oettinger (2013) A. G. Oettinger, “Automatic language translation,” in _Automatic Language Translation_. Harvard University Press, 2013.
  * Brown et al. (1988) P. F. Brown, J. Cocke, S. A. Della Pietra, V. J. Della Pietra, F. Jelinek, R. L. Mercer, and P. Roossin, “A statistical approach to language translation,” in _Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics_. John von Neumann Society for Computing Sciences, Budapest, 1988.
  * De Lange et al. (2021) M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 7, pp. 3366–3385, 2021.
  * Hadsell et al. (2020) R. Hadsell, D. Rao, A. A. Rusu, and R. Pascanu, “Embracing change: Continual learning in deep neural networks,” _Trends in cognitive sciences_ , vol. 24, no. 12, pp. 1028–1040, 2020.
  * Turc et al. (2019) I. Turc, M. Chang, K. Lee, and K. Toutanova, “Well-read students learn better: The impact of student initialization on knowledge distillation,” _CoRR_ , vol. abs/1908.08962, 2019. [Online]. Available: http://arxiv.org/abs/1908.08962
  * Tsai et al. (2019) H. Tsai, J. Riesa, M. Johnson, N. Arivazhagan, X. Li, and A. Archer, “Small and practical BERT models for sequence labeling,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Association for Computational Linguistics, 2019, pp. 3630–3634. [Online]. Available: https://doi.org/10.18653/v1/D19-1374
  * Tang et al. (2019) R. Tang, Y. Lu, L. Liu, L. Mou, O. Vechtomova, and J. Lin, “Distilling task-specific knowledge from BERT into simple neural networks,” _CoRR_ , vol. abs/1903.12136, 2019. [Online]. Available: http://arxiv.org/abs/1903.12136
  * Clark et al. (2019) K. Clark, M. Luong, U. Khandelwal, C. D. Manning, and Q. V. Le, “Bam! born-again multi-task networks for natural language understanding,” in _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , A. Korhonen, D. R. Traum, and L. Màrquez, Eds. Association for Computational Linguistics, 2019, pp. 5931–5937. [Online]. Available: https://doi.org/10.18653/v1/p19-1595
  * Liu et al. (2020) X. Liu, Y. Wang, J. Ji, H. Cheng, X. Zhu, E. Awa, P. He, W. Chen, H. Poon, G. Cao, and J. Gao, “The microsoft toolkit of multi-task deep neural networks for natural language understanding,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020_ , A. Celikyilmaz and T. Wen, Eds. Association for Computational Linguistics, 2020, pp. 118–126. [Online]. Available: https://doi.org/10.18653/v1/2020.acl-demos.16
  * Sun et al. (2019) S. Sun, Y. Cheng, Z. Gan, and J. Liu, “Patient knowledge distillation for BERT model compression,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Association for Computational Linguistics, 2019, pp. 4322–4331. [Online]. Available: https://doi.org/10.18653/v1/D19-1441
  * Iandola et al. (2020) F. N. Iandola, A. E. Shaw, R. Krishna, and K. Keutzer, “Squeezebert: What can computer vision teach NLP about efficient neural networks?” in _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, SustaiNLP@EMNLP 2020, Online, November 20, 2020_ , N. S. Moosavi, A. Fan, V. Shwartz, G. Glavas, S. R. Joty, A. Wang, and T. Wolf, Eds. Association for Computational Linguistics, 2020, pp. 124–135. [Online]. Available: https://doi.org/10.18653/v1/2020.sustainlp-1.17
  * Sun et al. (2020) Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou, “Mobilebert: a compact task-agnostic BERT for resource-limited devices,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics, 2020, pp. 2158–2170. [Online]. Available: https://doi.org/10.18653/v1/2020.acl-main.195
  * Zafrir et al. (2019) O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8BERT: quantized 8bit BERT,” in _Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition, EMC2@NeurIPS 2019, Vancouver, Canada, December 13, 2019_. IEEE, 2019, pp. 36–39. [Online]. Available: https://doi.org/10.1109/EMC2-NIPS53020.2019.00016
  * Lan et al. (2020) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=H1eA7AEtvS
  * Tambe et al. (2021) T. Tambe, C. Hooper, L. Pentecost, T. Jia, E. Yang, M. Donato, V. Sanh, P. N. Whatmough, A. M. Rush, D. Brooks, and G. Wei, “Edgebert: Sentence-level energy optimizations for latency-aware multi-task NLP inference,” in _MICRO ’21: 54th Annual IEEE/ACM International Symposium on Microarchitecture, Virtual Event, Greece, October 18-22, 2021_. ACM, 2021, pp. 830–844. [Online]. Available: https://doi.org/10.1145/3466752.3480095
  * Megalingam et al. (2019) R. K. Megalingam, R. S. Reddy, Y. Jahnavi, and M. Motheram, “Ros based control of robot using voice recognition,” in _2019 Third International Conference on Inventive Systems and Control (ICISC)_ , 2019, pp. 501–507.
  * Zhang et al. (2018) Y. Zhang, Z. Lu, C. Wang, C. Liu, and Y. Wang, “Voice control dual arm robot based on ros system.” 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR), 2018, pp. 232–237.
  * Megalingam et al. (2021) R. K. Megalingam, A. H. Kota, and V. K. T. P., “Optimal approach to speech recognition with ros,” in _2021 6th International Conference on Communication and Electronics Systems (ICCES)_ , 2021, pp. 111–116.
  * Zhang and Xu (2015) Y. Zhang and S. C. Xu, “Ros based voice-control navigation of intelligent wheelchair,” in _Engineering Decisions for Industrial Development_ , ser. Applied Mechanics and Materials, vol. 733. Trans Tech Publications Ltd, 4 2015, pp. 740–744.
  * Sharan et al. (2019) S. Sharan, T. Q. Nguyen, P. Nauth, and R. Araujo, “Implementation and testing of voice control in a mobile robot for navigation,” in _2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)_. IEEE, 2019, pp. 145–150.
  * Verma et al. (2020) A. Verma, D. Kumar, H. Maurya, A. Kumar, and M. P. Dwivedi, “Voice control robot using arduino,” _International Research Journal of Modernization in Engineering Technology and Science_ , vol. 2, p. 04, 2020.
  * Lv et al. (2008) X. Lv, M. Zhang, and H. Li, “Robot control based on voice command,” in _2008 IEEE International Conference on Automation and Logistics_. IEEE, 2008, pp. 2490–2494.
  * House et al. (2009) B. House, J. Malkin, and J. A. Bilmes, “The voicebot: a voice controlled robot arm,” in _Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, Boston, MA, USA, April 4-9, 2009_ , D. R. O. Jr., R. B. Arthur, K. Hinckley, M. R. Morris, S. E. Hudson, and S. Greenberg, Eds. ACM, 2009, pp. 183–192. [Online]. Available: https://doi.org/10.1145/1518701.1518731
  * Bilmes et al. (2005) J. A. Bilmes, X. Li, J. Malkin, K. Kilanski, R. Wright, K. Kirchhoff, A. Subramanya, S. Harada, J. Landay, P. Dowden, and H. Chizeck, “The vocal joystick: A voice-based human-computer interface for individuals with motor impairments,” in _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_. Vancouver, British Columbia, Canada: Association for Computational Linguistics, Oct. 2005, pp. 995–1002. [Online]. Available: https://aclanthology.org/H05-1125
  * Loper and Bird (2002) E. Loper and S. Bird, “Nltk: The natural language toolkit,” _arXiv preprint cs/0205028_ , 2002.
  * Manning et al. (2014) C. D. Manning, M. Surdeanu, J. Bauer, J. R. Finkel, S. Bethard, and D. McClosky, “The stanford corenlp natural language processing toolkit,” in _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_. The Association for Computer Linguistics, 2014, pp. 55–60.

## Appendix

##  Appendix A Dataset

The details of the datasets used in our evaluations are presented below.

###  A.1. Human Robot Interaction Corpus (HuRIC)

To utilize the HuRIC dataset in our task, we extracted intent information from
frame semantics. The original dataset666https://github.com/crux82/huric.
contains many intents that are closely related semantically; therefore, after
analyzing the commands, we merged several similar intents. For instance, based
on most of the commands in the dataset, we consolidated intent “Releasing” and
intent “Placing” into a single intent, namely “Placing”. The HuRIC dataset
consists of both simple and complex commands. To illustrate the difference,
consider commands like “Bring the book on the table in the kitchen,” which has
a single intent, “Bring”, while commands like “Please go to the kitchen and
inspect the sink” have two different intents: “Motion” and “Inspect”. However,
the dataset contains very few examples of complex commands.

###  A.2. GoEmotion

In contrast to the basic six emotions, which include only _one_ positive
emotion (joy), the GoEmotion taxonomy includes _twelve_ positive emotions.
Additionally, the taxonomy includes _eleven_ negative, _four_ ambiguous
emotion categories, and _one_ neutral emotion, making it widely suitable for
conversation understanding tasks that require a subtle differentiation between
emotion expressions.

The goal was to build a large dataset focused on conversational data, where
emotion is a critical communication component. Because the Reddit platform
offers a large, publicly available volume of content that includes direct
user-to-user conversation, it is a valuable resource for emotion analysis. So,
the authors built GoEmotion using Reddit comments from 2005 (the start of
Reddit) to January 2019, sourced from subreddits with at least 10K comments,
excluding deleted and non-English comments. The authors created a taxonomy to
maximize three objectives: (i) a greater coverage of the emotions expressed in
Reddit data, (ii) the coverage of types of emotional expressions, and (iii)
limit the overall number of emotions and their overlap. Such a taxonomy allows
data-driven, fine-grained emotion understanding while addressing potential
data sparsity for some emotions.

The authors iteratively defined and refined emotion label categories during
the data labeling stages, considering 56 emotion categories. They removed
emotions scarcely selected by raters, those with low inter-rater agreement due
to similarity to other emotions or difficulty detecting from the text.
Additionally, they incorporated emotions frequently suggested by raters and
were well-represented in the data. Finally, they optimized emotion category
names to maximize interpretability, resulting in a high inter-rater agreement,
with at least two raters concurring on at least one emotion label for 94% of
the examples.

###  A.3. CoNLL

We extracted ten days’ worth of data from the files representing the end of
August 1996 for the training and development set. The texts used for the test
set were from December 1996, while the pre-processed raw data covers the month
of September 1996. Each data file contains one word per line; empty lines
represent sentence boundaries. Additionally, a tag at the end of each line
indicates whether the current word is inside a named entity. The data
encompasses entities of four types: persons (PER), organizations (ORG),
locations (LOC), and miscellaneous names (MISC).

###  A.4. WNUT 2017

As discussed in 2.2.3, user comments were mined from different social media
platforms. Below, we provide some details about the dataset. The researchers
who proposed the dataset drew documents from various English-speaking
subreddits over January-March 2017. They selected these based on volume,
considering a variety of regions and granularities. For instance, they
included country- and city-level subreddits, along with non-geo-specific
forums such as /r/homeassistant, r/HomeImprovement, and /r/restaurants. The
documents were filtered to include only those between 20 and 400 characters.
Then, they split them into sentences and tagged them with the NLTK (Loper and
Bird, 2002) and Stanford CoreNLP (Manning et al., 2014).

The corpus contains comments from online video-sharing platforms (e.g.,
YouTube), specifically drawn from the all-time top 100 videos across all
categories within certain parts of the anglosphere. Researchers collected one
hundred top-level comments from each video, excluding non-English comments.
Twitter samples were collected from periods aligning with recent natural
disasters, specifically the Rigopiano avalanche and the Palm Sunday shootings.
The intention was to select content about emerging events that may contain
highly specific and novel toponyms. The dataset also includes another set of
user-generated content from StackExchange. Specifically, they downloaded title
posts and comments associated with five topics (movies, politics, physics,
sci-fi, and security) posted between January and May 2017 from archive.org.
Finally, they uniformly selected 400 samples from these title posts and
comments for each topic.

##  Appendix B Hardware Cost

Table 14 summarizes the hardware costs.

Hardware | Cost (USD)
---|---
Raspberry Pi 2 GB | 94
Raspberry Pi 4 GB | 109
Jetson 2 GB | 159
Jetson 4 GB | 219
UP2 2 GB | 179
UP2 4 GB | 199
UDOO (including RAM) | 610
UM25C energy meter | 28
Misc. (cables, memory cards) | 100
Total | $1697
Table 14. Hardware costs for setting up our experiments.