# AMSER: Adaptive Multimodal Sensing for Energy Efficient and Resilient
eHealth Systems

Emad Kasaeyan Naeini1, Sina Shahhosseini1, Anil Kanduri2, Pasi Liljeberg2,
Amir M. Rahmani1, Nikil Dutt1 1 Dept. of CS, University of California, Irvine,
USA, 2 Dept. of Computing, University of Turku, Finland
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

eHealth systems deliver critical digital healthcare and wellness services for
users by continuously monitoring physiological and contextual data. eHealth
applications use multi-modal machine learning kernels to analyze data from
different sensor modalities and automate decision-making. Noisy inputs and
motion artifacts during sensory data acquisition affect the i) prediction
accuracy and resilience of eHealth services and ii) energy efficiency in
processing garbage data. Monitoring raw sensory inputs to identify and drop
data and features from noisy modalities can improve prediction accuracy and
energy efficiency. We propose a closed-loop monitoring and control framework
for multi-modal eHealth applications, AMSER, that can mitigate garbage-in
garbage-out by i) monitoring input modalities, ii) analyzing raw input to
selectively drop noisy data and features, and iii) choosing appropriate
machine learning models that fit the configured data and feature vector - to
improve prediction accuracy and energy efficiency. We evaluate our AMSER
approach using multi-modal eHealth applications of pain assessment and stress
monitoring over different levels and types of noisy components incurred via
different sensor modalities. Our approach achieves up to 22% improvement in
prediction accuracy and 5.6$\times$ energy consumption reduction in the
sensing phase against the state-of-the-art multi-modal monitoring application.

##  I Introduction

eHealth applications deliver digital healthcare and wellness services for
users by continuously monitoring and analyzing physiological and contextal
sensory data [1]. eHealth applications feature: i) high volumes of input data
from multi-modal sensory devices, ii) intense machine learning algorithmic
computations for data analysis, and ii) limited energy and compute resources,
particularly at the sensing layer (e.g., wearable sensors) [2]. A large
category of eHealth applications are designed to monitor users in everyday
settings through wearable sensors, acquiring often noisy continuous
longitudinal data, compared to that of reliable clinical-grade sensors used in
clinical settings [3]. Input sensory data perturbations including noisy and
unreliable signal components, motion artifacts, and physical failure of
sensors challenges the resilience of eHealth applications [4]. Processing
perturbed sensory inputs affects prediction accuracy of machine learning
algorithms, while incurring significant computational and energy expense [5].

Multi-modal eHealth applications are executed in two phases viz., sensing \-
to acquire complementary data from multi-modal sensors for an application, and
sense-making \- to analyze the acquired data for predictive results [6]. Input
data perturbations at the multi-modal sensing phase affects the computational
workload and energy consumption in the subsequent sense-making phase.
Selective sensing through data filtering and pre-processing reduces the total
data volume to be sensed and processed at the sensing phase itself which
consequently leads to energy savings. Selective sensing extracts insightful
data to alleviate compute pressure, reduce energy consumption and improve
prediction accuracy in the sense-making phase. State-of-the-art multi-modal
applications use selective sensing techniques such as data filtering and
compression [7], context-aware sensing [8], and heterogeneity-aware sensing
[9] to reduce the total input data volume. Other techniques target model
optimization to reduce the compute intensity of sense-making algorithms [10].
Existing optimizations for multi-modal machine learning (MMML) based eHealth
applications address multi-modal sensing and sense-making independently [11,
12, 13]. Improving end-to-end system metrics of performance, energy
consumption, and prediction accuracy necessitates joint optimization of
sensing and sense-making phases. This exposes opportunities for i) adaptive
sensing: selectively extracting quality-related input data and features - to
alleviate compute pressure and reduce energy consumption incurred in
processing garbage and/or less insightful data, and ii) adaptive sense-making:
choosing suitable machine learning models by considering selective features
and modalities from the sensing phase - to improve prediction accuracy. Joint
optimization of multi-modal sensing and sense-making requires continuous
monitoring of sensor modalities to identify input data perturbations,
selective feature aggregation to isolate quality-related inputs, and choice of
learning algorithms suitable for the given sensor modalities and feature
vectors. To this end, we propose an adaptive multi-modal sensing and sense-
making approach, AMSER, for improving performance, energy consumption, and
prediction accuracy of eHealth applications. To demonstrate the potential of
our AMSER approach in improving both sensing and sense-making, we design a
proof-of-concept rule-based controller for adaptive feature selection, machine
learning model selection, and sensor configuration, based on the monitored
signal qualities. We integrate the controller into our framework running real
eHealth applications to guide the sensing and sense-making joint optimization
decisions in real-time for resilient, low-latency, and energy efficient
eHealth services. We summarize our contributions as follows:

  * •

Design of sensor-edge framework for multi-modal machine learning based eHealth
applications, capable of monitoring input signal quality, and detection of
discrepancies to guide sensing and sense-making optimization decisions.

  * •

Design of a proof-of-concept rule-based controller for adaptive feature
selection, modality selection, machine learning model selection, and sensor
configuration based on sensor modality monitoring - to improve performance,
energy, and prediction accuracy.

  * •

Demonstration of our framework’s efficiency on two eHealth case studies: pain
assessment and stress monitoring.

##  II Motivation

We illustrate the resilience challenges in MMML-based eHealth services and the
significance of adaptive sensing and control in addressing those challenges
through a motivational example of a pain monitoring application [14]. The pain
monitoring application acquires physiological data from different modalities
viz., Electrocardiography (ECG), Electrodermal Activity (EDA), and
Photoplethysmography (PPG) sensors to capture the autonomic nervous system
activity against pain. Figure 1 shows the pipeline composed of multi-modal
sensor data acquisition, feature aggregation, and machine learning model
selection - for training and prediction. In this example, sensors from each of
the ECG, EDA, and PPG modalities are sampled at 500Hz (2 channels), 4Hz, and
64 Hz, respectively. We extract relevant features from the raw input data and
aggregate the features from all the modalities into an early-fused feature
vector. We select a suitable machine learning model and train using the
aggregated features to predict the pain levels. The model yields an average
accuracy of 81% considering no noise on the input modalities. In practical
scenarios, sensory data from one or more modalities can be noisy, and/or have
data unavailability with perturbations such as motion artifacts, physical
damages and battery shutdown [2]. In this example, we describe the
implications of such sensory data perturbations on prediction accuracy and
energy consumption, using different approaches of application orchestration.
(a) Processing noisy data. Figure 1 (a) shows the baseline scenario of
processing sensory data without considering the noisy components. The ECG,
EDA, and PPG modalities generate data of 8KB, 32B, and 512B, with 52, 42, and
42 features, respectively. In this case, the data from the ECG modality has a
significant noisy component (e.g., due to improper patch contact to the
chest), while the EDA and PPG modalities (e.g., collected from the wrist)
generate quality inputs. Ignoring the input quality, features from all the
modalities are fused into a vector comprising 136 features. The model pool
comprises a set of machine learning models suitable for different feature
vectors and input modalities. From the model pool, a machine learning model
that suits the feature vector fused from 3 modalities (3-modal, 136 feature)
is chosen for training and prediction. This model yields an average accuracy
of 51%, with an end-to-end latency of 776 ms, and energy consumption of 5.14 J
and 3.85 J, for the edge and the sensor devices, respectively. The loss in
prediction accuracy can be attributed to the noisy data from the ECG modality
(garbage-in garbage-out effect). Further, this levies an unnecessary energy
consumption incurred in sensing and processing the noisy data.

Figure 1: Example experimental scenario demonstrating adaptive sensing
configuration for feature and modality selection. Figure 2: System
Architecture Overview.

(b) Processing with selective feature aggregation. Figure 1 (b) shows an
optimized scenario with selective feature aggregation for handling modalities
with noisy data. By considering the noisy components and aggregating only a
selected set of features from ECG modality, we improve prediction accuracy,
while lowering the computational effort and energy consumption. In this
scenario, the Selective feature aggregation selects 12 features from ECG
modality (shown in red box) among the original 52 full scale features, and
aggregates them with EDA and PPG features. The updated feature vector is sent
to the Model Selection to choose a machine learning model that suits the
feature vector for training and prediction. This model with selective feature
aggregation yields an average accuracy of 79%, which is 35% higher than that
of the baseline scenario with no input data considerations and only 2% less
than the baseline with the no-noisy modality condition (ideal condition).
Reduced features from the noisy ECG modality lowers the computational effort
and energy consumption. In this case, selective feature aggregation improved
the latency by 7% and energy consumption of the edge device by 16%. The
sensing energy consumption does not change in this scenario since sensing
configuration is unchanged.
(c) Processing with modality selection: Figure 1 (c) shows the optimized
scenario with modality selection for handling modalities with unavailable
data. In this scenario, data from the ECG modality is completely unavailable.
With data from the EDA and PPG modalities being available, features from these
modalities can be aggregated to build an updated feature vector. The Selective
feature aggregation aggregates 42 features from EDA and 42 features from PPG
modalities. From the Models Pool, the Model Selection chooses a machine
learning model that suits the updated feature vector with reduced number of
modalities and features (2-modal, 84 feature) for training and prediction.
This model yields an average accuracy of 74%, which is 31% higher than that of
the baseline scenario with no input data considerations. Note that reducing
the number of modalities to 2 (from 3 in Scenario (b)) lowers the accuracy
only by 7% (compared to the ideal scenario). Dropping the unavailable ECG
modality reduces the total data volume to be processed significantly. If the
information on the sense-making decision is fed back to the sensing layer to
adaptively change the sensing configuration, further energy saving at the
sensing layer is also possible, which is critical in battery-powered wearable
sensors. Overall, this leads to 46% improved latency, 50% lower energy
consumption at the edge device, and 53% lower energy consumption at the
sensors as compared to the baseline scenario of processing noisy data.

The example scenarios presented in Figure 1 demonstrates the advantages of
monitoring input modalities for selective feature aggregation, modality
selection, and model selection. However, this requires continuous monitoring
of input modalities, and intelligent control for selective aggregation,
modality, and model selection. To this end, we propose an automated framework
for adaptive multi-modal sensing to improve resilience and energy efficiency
of MMML-based eHealth applications.

##  III Adaptive Multimodal Sensing-SenseMaking Framework

We deploy a commonly used eHealth system architecture to implement our AMSER
approach, that consists of multiple sensor devices at the sensor layer and
computing resources at the edge layer. Figure 2 shows an overview of the
proposed framework. The sensor layer includes the multi-modal signal
capability needed for eHealth monitoring applications. The edge layer is a
computing device providing data gateway, data processing, signal monitoring,
adaptive control, and inference at the edge. We detail each level of the
framework below:

Data Gateway receives raw signal data from multi-modal sensors. The
physiological signals can potentially contain two main types of noises:
Baseline Wander (BW) and Motion Artifacts (MA).

Data Processing performs pre-processing and feature extraction with the data
collected from multi-modal sensors. Pre-processing module consists of data
synchronization of multi-modal signals, and bio-signal processing (e.g., peak
detection) from different modalities such as ECG, PPG, and EDA that are
essential for feature extraction, and finally filtering sensor modality inputs
to produce clean signals. Feature Extraction module extracts informative and
non-redundant values from the filtered signals. We extract time domain (mean,
standard deviation, rms) and frequency domain (power spectral density, median
frequency, central frequency) handcrafted features among the automatic
features using a variational autoencoder for all modalities [15]. This process
facilitates subsequent learning, leading to better interpretations for signal
quality assessments.

Signal Monitoring handles quality assurance of the data and features to be
used in the inference engine. This module observes the system parameters,
disruptive events, data quality, and control flow to analyze the situation and
context. Signal Monitoring assesses data and its extracted features quality by
monitoring key parameters from the sensing phase to identify events and
triggers for joint optimization of sensing and sense-making. For example, a
disruptive event in input data of a specific modality (e.g., motion artifacts
or sensor detachment) is a trigger at the sensing phase. In this regard, as
shown in Algorithm 1, signal monitoring assesses data ($D_{M}^{i}$) and its
extracted feature ($F_{M}^{i}$) quality and outputs the level of reliability
of each modality ($Rel_{M}^{i}$).

Signal quality: We consider three levels of signal quality viz., reliable,
noisy, and uncertain. Signal quality will assess the quality of sensing in
terms of electrodes attachment, signal-to-noise ratio (SNR), and motion
artifacts to determine the level of reliability. A signal with no noisy
components is labeled as reliable. Signals extracted from sensor modalities
whose electrodes are detached from the subject’s body and signals with an SNR
lower than a fixed threshold [16] are labeled as noisy. Signals with an SNR
above the fixed threshold yet below the acceptable levels are labeled as
uncertain.

Discrepancy Detector: The Discrepancy Detector acts as a second layer for
assessing the quality of signals that are labeled as reliable in the Signal
Quality phase by analyzing modality-specific parameters to identify any
discrepancies. Discrepancy Detector will recognize confounding factors in each
modality to identify levels of signal reliability. For instance, for ECG and
PPG modalities, we use the range of HR, RR interval (RRi) length, the ratio of
Max RRi to min RRi with their thresholds as the parameters to determine
discrepancies [16]. As another example, for EDA modality, we use low amplitude
and very low and steady tonic level for more than 1 minute as the discrepancy
detecting parameter [17, 18].

Data: $D_{M}^{i},F_{M}^{i},sensorStatus_{M}^{i}$

Result: $Rel_{M}^{i}$

1$Rel_{M}^{i}\leftarrow RELIABLE$;

2 if  _$sensorStatus_{M}^{i}==$ DETACHED_ then

3 $Rel_{M}^{i}\leftarrow NOISY$;

4

5else

6 if  _SNR $\leq$ threshold1_ then

7 $Rel_{M}^{i}\leftarrow NOISY$;

8

9 else

10 if  _threshold1 $\leq$ SNR $\leq$ threshold2_ then

11 $Rel_{M}^{i}\leftarrow UNCERTAIN$;

12

13

14

15

16for each modality $D_{M}^{i}$;

17 if  _Rules( $D_{M}^{i}$) $==$ PASS_ then

18 $Rel_{M}^{i}\leftarrow RELIABLE$;

19

20else

21 $Rel_{M}^{i}\leftarrow UNCERTAIN$;

22

23

Algorithm 1 Signal Monitoring Strategy.

Adaptive Controller analyzes inputs from the signal monitoring module to
adaptively select features and modalities, and configure sensing and
computation models. We store pre-trained models for different combinations of
signal modalities and aggregated feature vectors in a pool of models. The
adaptive controller is designed to holistically and dynamically control the
quality of sensing and sense-making, as shown in Algorithm 2. The adaptive
controller considers the reliability of each sensor modality ($Rel_{M}^{i}$),
the feature vector ($F_{M}^{i}$) extracted from Signal Monitoring, and the
model pool for feature aggregation and model selection. The controller module
adaptively selects the features that are relevant for prediction accuracy, and
a learning model that is suitable for the updated feature vector using a
simple rule based approach. In case a modality is labelled as uncertain, we
process the modality by dropping some of the less prominent features. If a
modality is noisy, we drop the input data from the modality entirely. The
controller will generate a signal to update the sensing configuration, such
that noisy sensor modalities can be turned off. Sensor modalities are turned
into the idle mode when the noise cannot be mitigated and thereby eliminating
the unnecessary computation and communication penalty. Finally, Model
Selection utilizes the selected features, and the current models available in
the Model Pool to load a preferable model for the Inference Engine taking into
consideration the sensing status and situation. Algorithm 2 is a simple proof-
of-concept rule-based policy to show the potentials of this approach. More
advanced algorithms (e.g., using reinforcement learning or fuzzy control) can
be devised to make intelligent decisions.

Data: $Rel_{M}^{i},F_{M}^{i},MP$

Result: Inference Model

1 $Sens_{M}^{i}\leftarrow ON$;

2 if  _$Rel_{M}^{i}==NOISY$_ then

3 $Sens_{M}^{i}\leftarrow OFF$;

4

5else

6 if  _$Rel_{M}^{i}==UNCERTAIN$_ then

7 $F_{M}^{i}\leftarrow$ Choose subset of $F_{M}^{i}$

8 else

9 $F_{M}^{i}\leftarrow F_{M}^{i}$

10

11if  _$Sens_{M}^{i}$ and $F_{M}^{i}$_ then

12 $Inf_{M}^{i}\leftarrow$ MP_M^i

Algorithm 2 Adaptive Controller.

##  IV Evaluation

We evaluate the proposed adaptive sensing framework through two case studies
of pain assessment [19, 20, 21] and stress monitoring [22]. Both case study
applications require continuous monitoring of multiple modalities including
ECG, EMG, PPG, and EDA signals, to detect pain and stress levels. Our
evaluation platform comprises a sensory node - to collect physiological
signals of ECG, EMG, PPG, and EDA from the subject, and an edge node - to
execute the inference. We implement the proposed method on a ODROID-XU3 with
an octa-core Exynos processor as the edge device. In the following, we detail
the two case studies and evaluate the case study applications’ accuracy,
energy efficiency, and performance using our AMSER proposed framework through
different scenarios with different types and levels of noisy and unreliable
input modalities. Further, we compare our AMSER approach against state-of-the-
art multi-modal pain monitoring application that use classification of
physiological signals [23] as the baseline ideal scenario.

###  IV-A Case Studies

We validate and demonstrate the efficacy of our AMSER approach through two
case studies from the affective computing domain: 1) pain assessment and 2)
stress monitoring.

####  IV-A1 Pain assessment

State-of-the-art pain assessment tools analyze changes in physiological
signals indicating different pain levels. In this work, we evaluate AMSER via
the pain assessment pre-trained models. These models are trained using
UCI_iHurtDB, a multi-modal dataset from postoperative patients in hospital
from 20 patients [24]. Data is collected from an eight-channel biopotential
acquisition system for EMG and ECG recording and Empatica E4 wristband for PPG
and EDA recording. We used the sensor modalities of 500 Hz ECG and EMG, 64 Hz
PPG, and 4 Hz EDA during our experiments. We pre-processed each modality and
segmented them into 60 second windows. For the prediction model to handle
potential noisy components in physiological signals, we augmented the raw
input training data with true types of noises such as BW and MA. Noisy
components induced in raw data are based on the characteristics of each
modality and extraction point such as chest, face, or wrist. Then, we extract
a combination of time domain, frequency domain and automatic features from
this window and trained using ML algorithms.

####  IV-A2 Stress Monitoring

The state-of-the-art for stress monitoring presents objective daily stress
measurements in everyday settings based on physiological signals [25, 26]. In
this work, we evaluate our proposed AMSER approach via stress monitoring
application based on pre-trained models using WESAD - a multi-modal dataset
featuring physiological and motion data recorded from both a wrist- and chest-
worn device of 15 subjects during a lab study [22]. Sensor modalities used
during our experiments are a 700 Hz ECG, 64 Hz PPG, and 4 Hz EDA. We then pre-
processed each modality and created 60second window segmentation. We augmented
true types of noises such as BW and MA based on the characteristics of each
modality. Then, we extracted a combination of time domain, frequency domain
and automatic features from this window and trained using ML algorithms.

###  IV-B Scenarios

We evaluated our proposed AMSER approach in 4 different scenarios with
different noise component levels for each case study. Table I summarizes
different scenarios (S1-S4), showing the type of noise induced in each
scenario. Scenario 1 is the ideal baseline case with no noise components,
Scenario 2 has uncertain signal components, Scenario 3 has one modality with
noisy component, and Scenario 4 has two noisy modalities. In Scenario 1, there
is no noise component, and thus no modality selection or feature selection is
applied. Scenario 2 comprises of baseline wandering noise in addition to the
original data for all the modalities. With the presence of noise, our proposed
AMSER approach selects specific significant features, reducing the total
number of features from the original feature vector. In Scenario 3, we add
additional motion artifacts on top of one modality while all the modalities
still have a baseline wander noise. In this scenario, the noisy modality is
completely unreliable. Thus, our proposed method will drop the modality and
uses a different model from Model Pool with 3 and 2 modalities for pain and
stress case studies, respectively. Scenario 4 comprises 2 modalities facing
severe motion artifact noises. In case of pain assessment, ECG and EMG
modalities are noisy with entirely unreliable data. Our proposed adaptive
sensing will drop both the ECG and EMG modalities and select a learning model
with 2 modalities for pain assessment application. In case of stress
monitoring, ECG and EDA modalities are noisy. Our proposed adaptive controller
drops both the ECG and EDA modalities and selects a learning model with the
single PPG modality for stress monitoring.

TABLE I: Experimental Scenarios for Pain and Stress applications. $+BW$, and
$+MA$ represent the presence of baseline wandering and motion artifact noise,
respectively.

|  | ECG | EMG | PPG | EDA
---|---|---|---|---|---
Pain | S1 | - | - | - | -
S2 | +BW | +BW | +BW | +BW
S3 | +BW | +BW | +BW+MA | +BW
S4 | +BW+MA | +BW+MA | +BW | +BW
Stress | S1 | - | N/A | - | -
S2 | +BW | N/A | +BW | +BW
S3 | +BW+MA | N/A | +BW+MA | +BW
S4 | +BW+MA | N/A | +BW | +BW+MA

We evaluate the accuracy and energy efficiency of both the pain monitoring
application (which estimates the pain intensity using 4 physiological signals
ECG, EMG, EDA, and PPG), and the stress monitoring application (which
calculates the stress level using 3 physiological signals ECG, PPG, and EDA)
using the iHurt_DB [24]. We trained different models for each levels of pain
intensities, varying in the types of modalities combined for the decision-
making system. After pre-processing, we extract a set of unique features from
each modality viz., 52 features of ECG, 42 features of EDA, and 42 features of
PPG [19, 20, 21, 26].

TABLE II: Decision made by AMSER during each scenario.

|  | ECG | EMG | PPG | EDA | Feature Vol (%)
---|---|---|---|---|---|---
Pain | S1 | ✓ | ✓ | ✓ | ✓ | 100
S2 | ✓ | ✓ | ✓ | ✓ | 63.8
S3 | ✓ | $\times$ | ✓ | ✓ | 35.8
S4 | $\times$ | $\times$ | ✓ | ✓ | 31.3
Stress | S1 | ✓ | N/A | ✓ | ✓ | 100
S2 | ✓ | N/A | ✓ | ✓ | 75.4
S3 | ✓ | N/A | ✓ | ✓ | 31.6
S4 | $\times$ | N/A | ✓ | ✓ | 14

###  IV-C Accuracy Evaluation

S1S2S3S4$0$$50$$100$Accuracy (%)AMSERBaseline ((a)) Pain

S1S2S3S4 ((b)) Stress

Figure 3: Accuracy analysis for AMSER vs. Baseline [23] for Pain and Stress
applications.

We compare the accuracy achieved by the model in each scenario with noisy
components (S2-S4) against the baseline ideal scenario $S1$ with no adaptive
modality and feature selection. Figure 3 shows the accuracy (in %) for pain
assessment and stress monitoring case studies under different scenarios. Our
proposed adaptive multimodal sensing technique provides an improved accuracy
in each of the scenarios for both the case studies, with a maximum gain of 22%
in S2 for pain assessment. In the presence of a lower level of noise i.e.,
uncertain signal, our proposed AMSER approach still utilizes the uncertain
modality with selected features, instead of either dropping or holding the
entire feature sets of the noisy modality. This approach leads to a 22% and
17% of accuracy improvement for pain and stress applications respectively, in
comparison with the baseline model which uses all the features from the noisy
modality. With a higher level noise i.e., noisy modality in scenario $S3$, our
proposed AMSER approach leads to 15% and 10% accuracy improvement for pain and
stress applications, respectively, compared to the baseline which uses all the
modalities. Lastly, scenario $S4$ shows that dropping 2 noisy modalities
results in better accuracy (13% and 11% improvement for pain and stress
applications, respectively), as compared to the baseline using noisy
modalities.

###  IV-D Efficiency and Performance Evaluation

We evaluate the efficiency and performance of AMSER in comparison with the
baseline. Figure 4 shows the energy efficiency improvement during four
scenarios for edge and sensor devices. We report the results for both Pain and
Stress applications. Figure 4 (a) shows the energy efficiency of the pain
assessment application. In Scenario $S4$, the energy gain is $2.19\times$ and
$5.63\times$ higher than the baseline for the edge and sensors, respectively.
In this case, AMSER provides $13.27\%$ better accuracy by dropping two noisy
modalities (See Figure 3 and Table II). Figure 4 (b) shows the energy
efficiency for the stress monitoring application. In Scenario $S3$, our
evaluation shows that AMSER improves the energy efficiency of the stress
monitoring application by $1.32\times$ and $2.63\times$ for the edge and
sensor devices, respectively (See Figure 4 (b)). In this case, AMSER improves
the accuracy and performance by $10.56\%$ and $27\%$, respectively (See Figure
3). Through the Scenario $S3$ for Stress application, AMSER keeps $31.6\%$ of
the features and drops the unreliable features (See Table II). Figure 5 (a)
shows the performance gains at the edge device for the pain assessment and
stress monitoring applications using our proposed AMSER approach, as compared
to the baseline Scenario $S1$. In Scenarios $S2$-$S4$, our proposed AMSER
approach reduces the number of features and/or drops input data from noisy
modalities. This lowers the total computational effort, leading to an increase
in performance, as compared to the baseline. Figure 5 (b) shows the data
volume transferred between the sensory devices and edge node using our
proposed AMSER approach, in comparison with the baseline. Our AMSER approach
adaptively selects features and/or drops noisy modalities, which reduces the
total input data volume significantly. For instance, in Scenario $S4$, two
noisy modalities are entirely dropped, leading to more than 6$\times$
reduction in transmitted data volume. Such reduction in data volume to be
transmitted further reduces the communication latency and energy expense
incurred in transmitting noisy data.

S1S2S3S4$0$$2$$4$$6$ Energy Eff (Baseline=1) AMSER-EdgeAMSER-Sensor ((a)) Pain

S1S2S3S4 ((b)) Stress

Figure 4: Energy efficiency analysis of the edge and sensor device for AMSER
vs. Baseline [23] for Pain and Stress applications.

S1S2S3S4$0$$1$$2$Speedup (Baseline=1)PainStress ((a))

S1S2S3S4$0$$2$$4$$6$Data Reduction ((b))

Figure 5: (a) Performance analysis of the edge device for AMSER vs. Baseline
[23] for Pain and Stress applications. (b) Data volume transferred between the
sensors and edge for AMSER vs. Baseline.

##  V Conclusion

We proposed AMSER, a novel adaptive, closed-loop multi-modal sensing framework
for energy efficient and resilient eHealth applications. AMSER guides sensing
and sense-making by monitoring input signal quality and discrepancy detection
via an intelligent control framework that reduces garbage data to achieve both
energy efficiency as well as improved quality. We demonstrated the AMSER
framework’s efficiency with two eHealth case studies on pain assessment and
stress detection, achieving up to $2.2\times$ and $5.6\times$ performance and
energy gains respectively, compared to the baseline with no optimization,
while achieving up to $22\%$ better accuracy. Our future work will focus on
devising more intelligent resource management policies for adaptive control
and co-optimization of both sensing and sense-making.

## References

  * [1] Colin O’Keeffe et al.  Role of ambulance response times in the survival of patients with out-of-hospital cardiac arrest.  EMJ, 2011.
  * [2] Emad Kasaeyan Naeini et al.  An edge-assisted and smart system for real-time pain monitoring.  In CHASE, 2019.
  * [3] G. Yang et al.  A health-iot platform based on the integration of intelligent packaging, unobtrusive bio-sensor, and intelligent medicine box.  IEEE Transactions on Industrial Informatics, 2014.
  * [4] Emad Kasaeyan Naeini et al.  A real-time ppg quality assessment approach for healthcare internet-of-things.  ANT, 2019.
  * [5] Ghulam M. et al.  A comprehensive survey on multimodal medical signals fusion for smart healthcare systems.  Information Fusion, 2021.
  * [6] Jiquan Ngiam et al.  Multimodal deep learning.  In ICML, 2011.
  * [7] Meenu Rani et al.  A systematic review of compressive sensing: Concepts, implementations and applications.  IEEE Access, 2018.
  * [8] Charith Perera et al.  Context aware computing for the internet of things: A survey.  IEEE communications surveys & tutorials, 2013.
  * [9] Sharief MA Oteafy.  A framework for heterogeneous sensing in big sensed data.  In GLOBECOM, 2016.
  * [10] D. Amiri et al.  Context-aware sensing via dynamic programming for edge-assisted wearable systems.  ACM HEALTH, 2020.
  * [11] Masinelli et al.  Self-aware machine learning for multimodal workload monitoring during manual labor on edge wearable sensors.  Design&Test, 2020.
  * [12] Murchana Baruah and Bonny Banerjee.  Modality selection for classification on time-series data.  In MileTS, 2020.
  * [13] Sicong Liu et al.  On-demand deep model compression for mobile devices: A usage-driven model selection framework.  In MobiSys, 2018.
  * [14] Philipp Werner et al.  Automatic recognition methods supporting pain assessment: A survey.  IEEE Transactions on Affective Computing, 2019.
  * [15] Aqajari et al.  pyeda: An open-source python toolkit for pre-processing and feature extraction of electrodermal activity.  ANT, 2021.
  * [16] C. Orphanidou et al.  Signal-quality indices for the ecg and ppg: Derivation and applications to wireless monitoring.  JBHI, 2015.
  * [17] Mathias Benedek et al.  A continuous measure of phasic electrodermal activity.  Journal of neuroscience methods, 2010.
  * [18] Hugo F. Posada-Quintero et al.  Electrodermal activity is sensitive to cognitive stress under water.  Frontiers in Physiology, 2018.
  * [19] Naeini et al.  Pain recognition with electrocardiographic features in postoperative patients: Method validation study.  JMIR, 2021.
  * [20] Aqajari et al.  Pain assessment tool with electrodermal activity for postoperative patients: Method validation study.  JMU, 2021.
  * [21] Rui Cao et al.  Objective pain assessment using wrist-based ppg signals: A respiratory rate based method.  In EMBC, 2021.
  * [22] Philip Schmidt et al.  Introducing wesad, a multimodal dataset for wearable stress and affect detection.  In ICMI, 2018.
  * [23] M. Kächele et al.  Multimodal data fusion for person-independent, continuous estimation of pain intensity.  In EANN, 2015.
  * [24] Naeini et al.  Prospective study evaluating a pain assessment tool in a postoperative environment: Protocol for algorithm testing and enhancement.  JRP, 2020.
  * [25] Victoriano Montesinos et al.  Multi-modal acute stress recognition using off-the-shelf wearable devices.  In EMBC, 2019.
  * [26] A. Tazarv et al.  Personalized stress monitoring using wearable sensors in everyday settings.  arXiv, 2021.


* Rehurek and Sojka [2011] Radim Rehurek and Petr Sojka.  Gensim–python framework for vector space modelling.  _NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic_ , 3(2), 2011.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.  URL https://arxiv.org/abs/1908.10084.
  * Robertson and Zaragoza [2009] Stephen Robertson and Hugo Zaragoza.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389, apr 2009.  ISSN 1554-0669.  doi: 10.1561/1500000019.  URL https://doi.org/10.1561/1500000019.
  * Sanh et al. [2020] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
  * Scao et al. [2022] Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ili’c, Daniel Hesslow, Roman Castagn’e, Alexandra Sasha Luccioni, Franccois Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Rose Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurenccon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa Etxabe, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris C. Emezue, Christopher Klamm, Colin Leong, Daniel Alexander van Strien, David Ifeoluwa Adelani, Dragomir R. Radev, Eduardo Gonz’alez Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady ElSahar, Hamza Benyamina, Hieu Trung Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Josephine L. Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro von Werra, Leon Weber, Long Phan, Loubna Ben Allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, Mar’ia Grandury, Mario vSavsko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad Ali Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla A. Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto L’opez, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, S. Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal V. Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Févry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiang Tang, Zheng Xin Yong, Zhiqing Sun, Shaked Brody, Y Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre Franccois Lavall’ee, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aur’elie N’ev’eol, Charles Lovering, Daniel H Garrette, Deepak R. Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Xiangru Tang, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, S. Osher Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenvek Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ananda Santa Rosa Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Olusola Ajibade, Bharat Kumar Saxena, Carlos Muñoz Ferrandis, Danish Contractor, David M. Lansky, Davis David, Douwe Kiela, Duong Anh Nguyen, Edward Tan, Emily Baylor, Ezinwanne Ozoani, Fatim T Mirza, Frankline Ononiwu, Habib Rezanejad, H.A. Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jan Passmore, Joshua Seltzer, Julio Bonis Sanz, Karen Fort, Lívia Macedo Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, M. K. K. Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nourhan Fahmy, Olanrewaju Samuel, Ran An, R. P. Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas L. Wang, Sourav Roy, Sylvain Viguier, Thanh-Cong Le, Tobi Oyebade, Trieu Nguyen Hai Le, Yoyo Yang, Zachary Kyle Nguyen, Abhinav Ramesh Kashyap, A. Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Kumar Singh, Benjamin Beilharz, Bo Wang, Caio Matheus Fonseca de Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel Le’on Perin’an, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Iman I.B. Bello, Isha Dash, Ji Soo Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthi Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, María Andrea Castillo, Marianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Cullan, Michael Weinberg, M Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patricia Haller, R. Chandrasekhar, R. Eisenberg, Robert Martin, Rodrigo L. Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Pratap Bharati, T. A. Laud, Th’eo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yashasvi Bajaj, Y. Venkatraman, Yifan Xu, Ying Xu, Yun chao Xu, Zhee Xao Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf.  Bloom: A 176b-parameter open-access multilingual language model.  _ArXiv_ , abs/2211.05100, 2022.
  * Schuster and Nakajima [2012] Mike Schuster and Kaisuke Nakajima.  Japanese and korean voice search.  _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5149–5152, 2012.
  * Schwartz et al. [2020] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni.  Green ai.  _Commun. ACM_ , 63(12):54–63, nov 2020.  ISSN 0001-0782.  doi: 10.1145/3381831.  URL https://doi.org/10.1145/3381831.
  * Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur Parikh.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.704.  URL https://aclanthology.org/2020.acl-main.704.
  * Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1162.  URL https://aclanthology.org/P16-1162.
  * Severyn and Moschitti [2015] Aliaksei Severyn and Alessandro Moschitti.  Learning to rank short text pairs with convolutional deep neural networks.  In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’15, page 373–382, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450336215.  doi: 10.1145/2766462.2767738.  URL https://doi.org/10.1145/2766462.2767738.
  * Shaw et al. [2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 464–468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2074.  URL https://aclanthology.org/N18-2074.
  * Shen et al. [2017] Gehui Shen, Yunlun Yang, and Zhi-Hong Deng.  Inter-weighted alignment network for sentence pair modeling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1179–1189, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1122.  URL https://aclanthology.org/D17-1122.
  * Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://aclanthology.org/D13-1170.
  * Soldaini and Moschitti [2020] Luca Soldaini and Alessandro Moschitti.  The cascade transformer: an application for efficient answer sentence selection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5697–5708, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.504.  URL https://aclanthology.org/2020.acl-main.504.
  * Stammbach and Ash [2020-10] Dominik Stammbach and Elliott Ash.  e-fever: Explanations and summaries for automated fact checking.  In Emiliano De Cristofaro and Preslav Nakov, editors, _Proceedings of the 2020 Truth and Trust Online (TTO 2020)_ , pages 32 – 43, Arlington, VA, 2020-10. Hacks Hackers.  ISBN 978-1-7359904-0-8.  doi: 10.3929/ethz-b-000453826.  Conference for Truth and Trust Online (TTO 2020) (virtual); Conference Location: online; Conference Date: October 16-17, 2020; Due to the Coronavirus (COVID-19) the conference was conducted virtually.
  * Stammbach and Neumann [2019] Dominik Stammbach and Guenter Neumann.  Team DOMLIN: Exploiting evidence enhancement for the FEVER shared task.  In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_ , pages 105–109, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-6616.  URL https://aclanthology.org/D19-6616.
  * Strubell et al. [2019] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1355.  URL https://aclanthology.org/P19-1355.
  * Sun et al. [2022] Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuanjing Huang.  BERTScore is unfair: On social bias in language model-based metrics for text generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3726–3739, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.245.  URL https://aclanthology.org/2022.emnlp-main.245.
  * Tan et al. [2018] Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Bowen Du, Weifeng Lv, and Ming Zhou.  Context-aware answer sentence selection with hierarchical gated recurrent neural networks.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 26:540–549, 2018.
  * Tay et al. [2020] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey, 2020.
  * Thorne et al. [2018] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1074.  URL https://aclanthology.org/N18-1074.
  * Tieleman et al. [2012] Tijmen Tieleman, Geoffrey Hinton, et al.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ , 4(2):26–31, 2012.
  * Torres-Moreno [2014] Juan-Manuel Torres-Moreno.  _Automatic Text Summarization_.  10 2014.  ISBN 978-1-84821-668-6.  doi: 10.1002/9781119004752.ch1.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Tran et al. [2018] Quan Hung Tran, Tuan Lai, Gholamreza Haffari, Ingrid Zukerman, Trung Bui, and Hung Bui.  The context-dependent additive recurrent neural net.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1274–1283, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1115.  URL https://aclanthology.org/N18-1115.
  * Trischler et al. [2017] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada, August 2017. Association for Computational Linguistics.  doi: 10.18653/v1/W17-2623.  URL https://aclanthology.org/W17-2623.
  * Turc et al. [2019] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Well-read students learn better: On the importance of pre-training compact models, 2019.
  * Tymoshenko and Moschitti [2021] Kateryna Tymoshenko and Alessandro Moschitti.  Strong and light baseline models for fact-checking joint inference.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4824–4830, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-acl.426.  URL https://aclanthology.org/2021.findings-acl.426.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Vinokourov et al. [2002] Alexei Vinokourov, Nello Cristianini, and John Shawe-Taylor.  Inferring a semantic representation of text via cross-language correlation analysis.  In S. Becker, S. Thrun, and K. Obermayer, editors, _Advances in Neural Information Processing Systems_ , volume 15. MIT Press, 2002.  URL https://proceedings.neurips.cc/paper/2002/file/d5e2fbef30a4eb668a203060ec8e5eef-Paper.pdf.
  * Voorhees and Tice [2000] Ellen M. Voorhees and Dawn M. Tice.  The TREC-8 question answering track.  In _Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00)_ , Athens, Greece, May 2000. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2000/pdf/26.pdf.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5446.  URL https://aclanthology.org/W18-5446.
  * Wang et al. [2019a] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _CoRR_ , abs/1905.00537, 2019a.  URL http://arxiv.org/abs/1905.00537.
  * Wang et al. [2007] Mengqiu Wang, Noah A. Smith, and Teruko Mitamura.  What is the Jeopardy model? a quasi-synchronous grammar for QA.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 22–32, Prague, Czech Republic, June 2007\. Association for Computational Linguistics.  URL https://aclanthology.org/D07-1003.
  * Wang and Jiang [2017] Shuohang Wang and Jing Jiang.  A compare-aggregate model for matching text sequences.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=HJTzHtqee.
  * Wang et al. [2019b] Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si.  Structbert: Incorporating language structures into pre-training for deep language understanding, 2019b.
  * Wang [2023] Xinyue Wang.  Large web archive collection infrastructure and services, 2023.  URL http://hdl.handle.net/10919/113345.
  * Warstadt et al. [2019] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641, 2019.  doi: 10.1162/tacl˙a˙00290.  URL https://aclanthology.org/Q19-1040.
  * West et al. [2007] Jeremy West, Dan Ventura, and Sean Warnick.  A theoretical foundation for inductive transfer.  Talk, 2007.  Department of Computer Science, Faculty Advisor: Dan Ventura.
  * Widenius et al. [2002] Michael Widenius, Davis Axmark, and Paul DuBois.  _Mysql Reference Manual_.  O’Reilly & Associates, Inc., USA, 1st edition, 2002.  ISBN 0596002653.
  * Wikipedia contributors [2004] Wikipedia contributors.  Natural language processing — Wikipedia, the free encyclopedia, 2004\.  URL https://en.wikipedia.org/wiki/Natural_language_processing.  [Online; accessed 22-July-2004].
  * Williams et al. [2018] Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1101.  URL https://aclanthology.org/N18-1101.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Xu et al. [2023] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023.
  * Xu et al. [2021] Jingjing Xu, Wangchunshu Zhou, Zhiyi Fu, Hao Zhou, and Lei Li.  A survey on green deep learning.  _ArXiv_ , abs/2111.05193, 2021.
  * Yang et al. [2015] Yi Yang, Wen-tau Yih, and Christopher Meek.  WikiQA: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2013–2018, Lisbon, Portugal, September 2015\. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1237.  URL https://aclanthology.org/D15-1237.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf.
  * Yoon et al. [2019] Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung.  A compare-aggregate model with latent clustering for answer selection.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , CIKM ’19, page 2093–2096, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450369763.  doi: 10.1145/3357384.3358148.  URL https://doi.org/10.1145/3357384.3358148.
  * Zaheer et al. [2020] Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big bird: Transformers for longer sequences.  _CoRR_ , abs/2007.14062, 2020.  URL https://arxiv.org/abs/2007.14062.
  * Zarharan et al. [2021] Majid Zarharan, Mahsa Ghaderan, Amin Pourdabiri, Zahra Sayedi, Behrouz Minaei-Bidgoli, Sauleh Eetemadi, and Mohammad Taher Pilehvar.  ParsFEVER: a dataset for Farsi fact extraction and verification.  In _Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics_ , pages 99–104, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.starsem-1.9.  URL https://aclanthology.org/2021.starsem-1.9.
  * Zhang et al. [2020a] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2020a.
  * Zhang et al. [2020b] Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward.  Multi-stage pre-training for low-resource domain adaptation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5461–5468, Online, November 2020b. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.440.  URL https://aclanthology.org/2020.emnlp-main.440.
  * Zhang et al. [2022a] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022a.
  * Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert, 2019.  URL https://arxiv.org/abs/1904.09675.
  * Zhang et al. [2022b] Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei.  Momentum calibration for text generation, 2022b.
  * Zhang et al. [2021] Zeyu Zhang, Thuy Vu, and Alessandro Moschitti.  Joint models for answer verification in question answering systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3252–3262, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.252.  URL https://aclanthology.org/2021.acl-long.252.
  * Zhao et al. [2020] Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary.  Transformer-xh: Multi-evidence reasoning with extra hop attention.  In _The Eighth International Conference on Learning Representations (ICLR 2020)_ , April 2020.  URL https://www.microsoft.com/en-us/research/publication/transformer-xh-multi-evidence-reasoning-with-extra-hop-attention/.
  * Zhong et al. [2020] Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin.  Reasoning over semantic-level graph for fact checking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6170–6180, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.549.  URL https://aclanthology.org/2020.acl-main.549.
  * Zhou et al. [2019] Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  GEAR: Graph-based evidence aggregating and reasoning for fact verification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 892–901, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1085.  URL https://aclanthology.org/P19-1085.
  * Zhu et al. [2015] Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, 2015.

##  Appendix A Background & Related Work

###  A.1 Data Organization

###### Structured data

Data are structured when they are defined as a set of entities/concepts and
relations between them. Some of the oldest and most common techniques to store
structured data are tables and relational databases, such as MySQL [Widenius
et al., 2002] or MariaDB111https://mariadb.org. Those databases are usually
queried in a specific language (e.g. SQL) to extract interesting entities and
relations.

Other relevant techniques to store structured data are formal ontologies and
knowledge graphs. Ontologies are representations of a domain through the usage
of categories, entities and relations. The goal of an ontology is to represent
a domain with all its characteristics and properties. Those structures are
usually represented with graphs or databases.

On the other hand, knowledge graphs are similar to ontologies but more data-
driven, and the representation of data may change based on the scope. When
building a knowledge graph, the objective is to allow users to find target
concepts and related entities as quickly as possible. Those structures are
usually backed by a graph database (GDB).

###### Semi-structured data

Data are semi-structured when they are organized similarly to structured data
but attributes are variable and there is no general schema that defines
concepts and entities. Moreover, entities and relations may contain attributes
that relate to unstructured data. An example of a database to store semi-
structured data is MongoDB222https://www.mongodb.com, in which the user is
allowed to store custom dictionaries of key-value objects. Another example of
semi-structured data is Wikipedia: the entities are the different arguments
described by every page and relations are the links between different pages.

The growth of the World Wide Web is increasing exponentially the amount of
semi-structured data available for Information Retrieval. Most of the data on
the WWW is semi-structured because each blog, newspaper or website uses a
different organization schema, or doesn’t use one at all. Today, most of the
algorithms based on neural networks for NLP are first pre-trained on huge
amounts of semi-structured data before being specialized on downstream
applications. For example, in our experiments, we create self-supervised
training functions exploiting the weak structure of large corpora, which only
define a paragraph-level subdivision.

###### Unstructured data

Unstructured data in NLP are represented by raw text, without any kind of
supervision. Unstructured text can be seen as a sequence of words, numbers,
dates and other forms of free text. Recent advances in NLP allow the
conversion from unstructured to semi-structured data through methods that
search for structures inside the text, such as entity extraction algorithms
(NER) and pattern recognition systems.

###  A.2 Language Models

####  A.2.1 Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are a specialized type of neural network
suited for sequential data processing, including natural language, time
series, and audio. While RNNs share their origin with Feed-Forward neural
networks, they belong to a distinct category due to the potential presence of
cycles in the information flow graph.

When applied to text, RNNs create a sentence-level representation by
traversing the input text from left to right. RNNs consist of recurrent units,
where each unit’s output becomes the input for the same unit at the next time
step. This design enables the network to maintain an internal state (memory)
that can capture dependencies between elements in the input sequence. In
processing a sequence of elements, an RNN typically passes the input elements
through the network one at a time, using each output as the input for the next
time step. The network’s internal state is updated based on both the current
input and the previous internal state.

Training RNNs involves employing backpropagation through time, a variant of
the backpropagation algorithm used for standard Feed-Forward neural networks.
The network is unrolled over time to treat it as a standard Feed-Forward
neural network, allowing for gradient computation and weight updates through
gradient descent.

However, RNNs suffer from certain drawbacks, including inefficiency due to
their inability to process tokens in parallel and the vanishing information
problem for early tokens in long sequences [Kolen and Kremer, 2001].

####  A.2.2 Convolutional Neural Networks

Convolutional Neural Networks (CNNs) represent a type of Artificial Neural
Networks (ANNs) initially designed for Computer Vision tasks, including image
classification and object recognition, but they have also been successfully
adapted for NLP tasks.

In CNNs, convolution operates by extracting features from input data using a
set of learnable filters. These filters slide over the input data, performing
a dot product between the filter entries and the input data, generating a
feature map.

An important characteristic of CNNs is weight sharing, where the same filters
are utilized to scan the entire input. This feature reduces the number of
parameters the network needs to learn and improves efficiency. CNNs also
incorporate pooling layers and activation functions, which respectively
downsample the feature maps and introduce non-linearity into the model.

For NLP tasks, CNNs treat the input text as a 2D matrix, with each word
represented as a row and each word embedding represented as a column. Similar
to image recognition tasks, the filters in CNNs for NLP slide over the input
text, performing dot products between the filter entries and the input text,
resulting in feature maps.

A notable distinction between CNNs for NLP and image recognition lies in the
use of pooling. While pooling is employed in image recognition to downsample
feature maps, it is typically avoided in NLP tasks to prevent the loss of
vital information. Instead, max pooling is often employed to extract the most
salient features from the feature map.

One significant advantage of CNNs over RNNs is their efficiency in processing
input text in parallel, enabling them to take advantage of hardware
acceleration technologies for faster multiplication of large matrices.

###  A.3 Tokenization methodologies

Here we provide an in-depth analysis of the most common tokenization
techniques.

####  A.3.1 Byte-Pair Encoding

BPE is a tokenization algorithm that follows the philosophy of representing
frequent words with single tokens while splitting uncommon words in multiple
tokens. It is derived from the compression algorithm [Gage, 1994] that
iteratively replaces the most frequent pair of bytes with a new, unseen, byte.
In the BPE tokenizer, the vocabulary ${\mathbb{V}}$ is built by iteratively
merging the token pair that appears more frequently in the training data and
adding it to ${\mathbb{V}}$. Algorithm 1 provides more details about the
vocabulary creation procedure. This approach is bottom-up because it starts
from a small vocabulary ${\mathbb{V}}$ containing only single characters and
builds up new tokens through the concatenation of frequent pairs.

1:Training documents ${\mathbb{D}}$, vocabulary size $k$

2:function BPE(${\mathbb{D}}$, $k$)

3: ${\mathbb{V}}\leftarrow\text{characters}({\mathbb{D}})$ $\triangleright$
Initialize ${\mathbb{V}}$ with all the single characters in the training data

4: while $|{\mathbb{V}}|<k$ do

5:
$\\{((t_{i},t_{i+1}),c_{i})\\}_{i\in{\mathbb{I}}}\leftarrow\text{get\\_bigrams\\_count}({\mathbb{D}},{\mathbb{V}})$
$\triangleright$ Get bigrams counts splitting over ${\mathbb{V}}$

6: $j\leftarrow\operatorname*{arg\,max}_{i\in{\mathbb{I}}}c_{i}$
$\triangleright$ Select most frequent token pair

7: $t\leftarrow t_{j}\cdot t_{j+1}$ $\triangleright$ “$\cdot$” is the
concatenation operator

8: ${\mathbb{V}}\leftarrow{\mathbb{V}}\cup\\{t\\}$

9:
${\mathbb{D}}\leftarrow\text{replace\\_bigram\\_with\\_unigram}({\mathbb{D}},[t_{i},t_{i+1}],t)$

10: end while

11: return ${\mathbb{V}}$

12:end function

Algorithm 1 Example of vocabulary creation with BPE tokenizer.

BPE’s encoding and decoding are performed with a greedy algorithm. The
encoding procedure searches iteratively for the longest token
$t\in{\mathbb{V}}$ that matches the start of the given input sequence.
Usually, before the vocabulary creation input sentences are split over
whitespaces and punctuation to ensure no token crosses word boundaries.
Moreover, special characters are be prepended or appended to every input word
to indicate the start or the end. Regarding the decoding, it is a simple
concatenation of every token. The division in words by inserting whitespaces
is ensured to be correct by taking advantage of the additional special
characters inserted before the tokenization. The following is an example of
how the BPE tokenizer of RoBERTa [Liu et al., 2019b], which is a very
prominent language model, splits text into tokens:

“Harry Potter and the Philosopher’s Stone”
$\displaystyle{\mathmakebox[width("{}={}")][c]{{\Downarrow}}}$
$\displaystyle[\text{``Harry''},\text{``ĠPotter''},\text{``Ġand''},\text{``Ġthe''},\text{``ĠPhilos''},\text{``opher''},\text{``'s''},\text{``ĠStone''}]$

Notice how the word “Philosopher’s” will be decoded correctly thanks to the
special character “Ġ” use by RoBERTa to indicate the start of a new word in
the original text.

Among the advantages of BPE there is the ability to tokenize words that were
not seen at training time by only using the available tokens. In the worst
case, the new word will be split in single characters, which were part of the
initial vocabulary. BPE is used by many Transformer models such as GPT-2
[Radford et al., 2019], GPT-3 [Brown et al., 2020] and RoBERTa [Liu et al.,
2019b].

####  A.3.2 WordPiece

WordPiece is a tokenization algorithm derived from BPE in which the selection
of the token pair to merge is performed with a more sophisticated technique.
In particular, WordPiece does not select the symbol pairs that are most
commonly used, but rather the ones that will increase the likelihood of the
training data when added to the vocabulary.

This means that it calculates the likelihood of each symbol pair by dividing
the joint probability of the pair by the probability of the individual
symbols. The symbol pair with the highest ratio is then selected. This
approach is different from BPE, as WordPiece takes into account the potential
loss of information when merging two symbols to determine if it is a
worthwhile decision.

The algorithm to create the vocabulary ${\mathbb{V}}$ is the same as BPE
(Algorithm 1), after changing line 5 into:

$j\leftarrow\operatorname*{arg\,max}_{i\in{\mathbb{I}}}\frac{P(t_{i},t_{i+1})}{P(t_{i})P(t_{i+1})}$
(A.1)

One of the benefits of using WordPiece is that it has been found to be less
prone to over-segmenting words, and it has been shown to be effective when
used in neural Machine Translation models. WordPiece is used in well known
Transformer models such as BERT [Devlin et al., 2019], DistilBERT [Sanh et
al., 2020] and ELECTRA [Clark et al., 2020].

####  A.3.3 UnigramLM

UnigramLM is a top-down tokenization approach which iteratively shrinks the
vocabulary size by removing tokens. UnigramLM usually initializes the
vocabulary with all the words and their substrings in training data. Then,
given some loss function such as the log-likelihood, at each iteration the
algorithm trains a simple unigram language model over the input data
${\mathbb{D}}$ and selects a subset of tokens that, if removed, would increase
the loss over the data the lowest. In other words, at each iteration the
subset of tokens that influence the loss the less are removed. Common
implementations remove the about the 10% of the tokens at each iteration until
the vocabulary reaches the desired size. It is also a good practice to include
in the vocabulary all the single characters of a language to ensure every word
can be tokenized and without recurring to the OOV token. Algorithm 2 provides
an example of UnigramLM vocabulary creation.

1:Training documents ${\mathbb{D}}$, vocabulary size $k$, fraction $\alpha$ of
tokens to remove at each iteration

2:function UnigramLM(${\mathbb{D}}$, $k$)

3: ${\mathbb{V}}\leftarrow\text{words\\_and\\_subwords}({\mathbb{D}})$
$\triangleright$ Initialize ${\mathbb{V}}$ with words and subwords in the
training data

4: while $|{\mathbb{V}}|>k$ do

5:
$M\leftarrow\text{train\\_unigram\\_language\\_model}({\mathbb{D}},{\mathbb{V}})$
$\triangleright$ Train Unigram LM over ${\mathbb{D}}$

6: ${\mathbb{L}}=\\{\\}$ $\triangleright$ Will contain loss decrease for each
token

7: for $t\in{\mathbb{V}}$ do

8:
$M^{\prime}\leftarrow\text{train\\_unigram\\_language\\_model}({\mathbb{D}},{\mathbb{V}}\setminus\\{t\\})$

9: ${\mathbb{L}}_{t}\leftarrow
P_{M}({\mathbb{D}})-P_{M^{\prime}}({\mathbb{D}})$ $\triangleright$ Loss
difference by removing token $t$ from ${\mathbb{V}}$

10: end for

11: for $(|{\mathbb{V}}|\cdot\alpha)$ times do $\triangleright$ Remove the
$\alpha$ fraction of less influent tokens

12: $t=\operatorname*{arg\,min}_{i}{\mathbb{L}}_{i}$

13: ${\mathbb{V}}\leftarrow{\mathbb{V}}\setminus\\{t\\}$ $\triangleright$ Trim
vocabulary

14: ${\mathbb{L}}\leftarrow{\mathbb{L}}\setminus\\{{\mathbb{L}}_{t}\\}$
$\triangleright$ Avoid selecting again ${\mathbb{L}}_{t}$ from ${\mathbb{L}}$

15: end for

16: end while

17:
$M\leftarrow\text{train\\_unigram\\_language\\_model}({\mathbb{D}},{\mathbb{V}})$
$\triangleright$ Train final Unigram LM over ${\mathbb{D}}$

18: return ${\mathbb{V}},M$

19:end function

Algorithm 2 Example of vocabulary creation with UnigramLM tokenizer.

Since UnigramLM is based on vocabulary reduction and not on merging rules like
BPE and WordPiece, there may se several ways of tokenizing the same string.
This may hurt language models performance because different tokenization of
the same string may be provided to the model. For this reason, an UnigramLM
tokenizer saves also the unigram language model over the final vocabulary, see
line 18 of Algorithm 2. The final language model $M$ is then used to compute
the likelihood of every possible tokenization. In practice the most likely
tokenization is chosen to have a deterministic split of text in subwords.

UnigramLM alone is not used by any relevant language model. However, it’s
vocabulary creation technique is often implemented along with Sentence-Piece
tokenization.

####  A.3.4 Sentence-Piece

Sentence-Piece is not a new tokenization technique by itself, but rather a
variant of the previously discussed algorithms in which the input text is not
split over whitespaces to separate words. By treating the whitespaces between
words as all the other characters, Sentence-Piece can be trained on streams of
raw text in several languages without the need of ad-hoc adaptations. For
example, it can successfully tokenize Chinese of Japanese text, which does not
contain whitespaces, as well as English or German.

The vocabulary creation of a Sentence-Piece tokenizer can be based on either
Byte-Pair Encoding or UnigramLM. Thus, it can tokenize almost any text without
producing OOV tokens. Sentence-Piece is used in many start-of-the-art language
models such as ALBERT [Lan et al., 2020], XLNet [Yang et al., 2019], DeBERTaV2
[He et al., 2020], DeBERTaV3 [He et al., 2021] and Google T5 [Raffel et al.,
2020].

####  A.3.5 Loss functions, Back-Propagation and Optimization

During the pre-training phase of language models, the primary objective is
typically framed as a classification task. With token-level objectives, the
model is trained to predict the original input by reconstructing it from
various forms of corruption applied during training. To achieve this, a
classification layer, such as a linear projection, is employed to map the
output embeddings of the model to different categorical labels, enabling it to
effectively classify and reconstruct the original input. In sentence-level
tasks and fine-tuning, the output embeddings of the model are exploited to
predict a categorical label that represents a feature of the whole input, such
as the opinion of a user in Sentiment Analysis, or the relation between
question and answer in Question Answering.

###### Loss function

The loss function is a metric that measures the accuracy of the predictions of
a model compared to the gold labels. Transformer models commonly use the
Cross-Entropy Loss function for both pre-training and fine-tuning stages. The
Cross-Entropy Loss measures the average number of bits required to identify an
event drawn from the distribution of the true labels $\mathcal{L}$ when using
a scheme optimized for the predicted probabilities distribution $\mathcal{P}$.
It provides a measure of the dissimilarity between these two distributions,
indicating how well the predicted probabilities align with the gold labels. In
the context of classification, it calculates the loss by comparing the
predicted class probabilities with the categorical labels. Given a label
$y\in{\mathbb{C}}$, where the latter is the set of possible classes, and a
vector of predictions over the classes $[p_{1},\dots,p_{|{\mathbb{C}}|}]$, the
Cross-Entropy loss can be computed as follows:

$L_{CE}({\bm{\mathrm{p}}},y)=-\sum_{i=1}^{|{\mathbb{C}}|}\mathbbm{1}_{\\{y=i\\}}\log\frac{e^{p_{i}}}{\sum_{i=0}^{|{\mathbb{C}}|}e^{p_{i}}}$
(A.2)

Moreover, the Cross-Entropy definition can be simplified when the task is
binary classification and the predictions are already softmaxed:

$L_{CE}(p,y)=-\big{(}y\log p+(1-y)\log(1-p)\big{)}$ (A.3)

Once the loss over the predictions and the labels is computed, the back-
propagation algorithm is used to compute the gradients for the parameters of
the model.

###### Back-propagation

Given the error in the predictions, the back-propagation algorithm computes
the gradients of all the parameters, enabling the model to update its weights
and improve its performance through gradient descent.

Mathematically, let’s consider a model $\mathcal{M}$ that is composed of a
sequence of parametric functions $f_{1},\dots,f_{n}$ which are applied in
order to an input ${\bm{\mathrm{x}}}$:

${\bm{\mathrm{p}}}=f_{n}\circ f_{n-1}\circ\dots\circ f_{1}({\bm{\mathrm{x}}})$
(A.4)

We call ${\bm{\mathrm{h}}}_{i}$ the output of the layer $f_{i}$. Given some
loss function, such as the Cross-Entropy defined before, and a target label
$y$, we can compute the error $L$ with some loss function, such as Cross-
Entropy:

$L=L_{CE}({\bm{\mathrm{p}}},y)$ (A.5)

Then, by applying the Chain Rule [Goodfellow et al., 2016], we can compute the
partial derivatives for each layer $f_{i}$ in the model $\mathcal{M}$. This
rule allows us to propagate gradients through multiple layers of a neural
network, by unrolling the gradients computed over the whole model to the
gradients of each layer. This is performed as follows:

$\nabla\mathcal{M}=\frac{\partial L}{\partial{\bm{\mathrm{x}}}}=\frac{\partial
L}{\partial f_{n}}\cdot\frac{\partial f_{n}}{\partial f_{n-1}}\cdot\ \cdots\
\cdot\frac{\partial f_{2}}{\partial f_{1}}\cdot\frac{\partial
f_{1}}{\partial{\bm{\mathrm{x}}}}$ (A.6)

Then, for each layer $f_{i}$, we can exploit the output of the previous layer
${\bm{\mathrm{h}}}_{i-1}$, the output of the current layer
${\bm{\mathrm{h}}}_{i}$ and the derivatives computed for the next layer
$\frac{\partial f_{i+1}}{\partial f_{i}}$ to compute the gradients of the
current layer parameters.

###### Optimization

By efficiently applying the Chain Rule, we can propagate the gradients
backward through the network, enabling the calculation of parameter updates
and optimization through gradient descent. This process allows the network to
learn and improve its performance by adjusting the weights based on the
computed gradients. Suppose you have a layer $i$ which is a parametric
function $f_{i}$ with weights ${\bm{W}}_{i}$ and gradients
$\nabla{\bm{W}}_{i}$, previously computed with back-propagation. The common
approach to update ${\bm{W}}_{i}$ is gradient descend:

${\bm{W}}_{i}^{\prime}={\bm{W}}_{i}-\eta\ \nabla{\bm{W}}_{i}$ (A.7)

where $\eta$ is called the learning rate and determines the speed at which the
model converges. The original Gradient Descent algorithm is no longer used due
to its sensitivity and instability while searching for the local minima.
Common optimizer algorithms such as Adam [Kingma and Ba, 2017], RMSProp
[Tieleman et al., 2012] and Adagrad [Lydia and Francis, 2019] improve over
Gradient Descend by storing additional data for each weight in the model. They
allow to maintain a more general direction toward the local minima while
optimizing, thus increasing the training stability and the model
generalization capabilities.

###  A.4 Machine Reading Comprehension Branches

####  A.4.1 Extractive Question Answering

Extractive Question Answering involves finding specific answers to questions
within a given text or set of texts. This is different from Generative (or
Abstractive) QA, which involves generating new text as an answer to a
question, rather than extracting an answer from existing text.

In Extractive QA, the system is given a question and a set of documents, and
is expected to identify the specific piece of text within the documents that
provides the answer to the question. This typically involves identifying the
relevant parts of the text, such as named entities or specific phrases, and
extracting them as the answer.

More formally, in Extractive QA a model $\mathbf{C}$ is given a question $q$
and a set of documents ${\mathbb{D}}$, which may contain answer spans $a_{i}$
in some $D_{i}\in{\mathbb{D}}$. The goal of the model is to extract the answer
spans $a_{i}$ from ${\mathbb{D}}$ given the question $q$. Typically, the
performance are measured by checking the degree of overlap between the gold
answer spans and those predicted by the model. The prediction of the answer
spans by the model $\mathbf{C}$ can be performed with different techniques.
For example, the model may output the start and end positions of the correct
answer or a binary classifier over each token of the documents in
${\mathbb{D}}$ could be used to predict the belonging to the answer.

####  A.4.2 Generative Question Answering

Generative Question Answering regards the generation of new text as an answer
to a question, rather than extracting an answer from existing documents. This
involves using Machine Learning techniques to learn how to generate coherent
and appropriate responses to questions, based on a large dataset of examples.
The goal is to generate correct answers that should be indistinguishable from
human-generated text. The input set of retrieved documents could be used as a
source of knowledge to help in the answer generation. However, some recent
models have such a large number of internal parameters that are able to
memorize very large quantities of information already while pre-training, and
thus can answer most questions without external knowledge [Brown et al., 2020;
Raffel et al., 2020].

In general, measuring the performance of generative models is a hard task, and
evaluating generated answers is no exception. The main problem is that the
model may output text with the same meaning of the gold answer but with a
different formulation and using different terms. Thus, the best evaluation of
generative model’s outputs is manual annotation by well-trained crowd-workers.
However, this method it is very expensive and the quality of the annotation is
subject to variations based on annotator’s salary, educational background and
geographical location.

As an alternative, automatic evaluation techniques can be applied to evaluate
generated answers. The most common metrics for automatic evaluation are BLEU
[Papineni et al., 2002] and ROUGE [Lin, 2004], which were developed for
Machine Translation and Summarization respectively. BLEU works by counting the
ratio of $n$-grams that overlap between the generated answer and the target
answer(s). On the other hand, ROUGE is an evaluation package containing
different metrics. ROUGE-N for example, is similar to BLEU and works by
counting common $n$-grams, however it is recall oriented and not precision-
based as BLEU. ROUGE-L searches instead for the Longest Common Subsequences
(LCS) between the generated answer and the gold references. Finally, we
mention ROUGE-S, which uses skip-bigram co-occurrence statistics to measure
the similarity between the generated string and the set of target references.
Since generated answers may contain the same meaning of the target reference
but expressed in a different form, the larger the number of gold references
for each question, the better the evaluation.

####  A.4.3 Abstractive Question Answering

Abstractive Question Answering, is a type of generative QA that involves
generating answers by summarizing or paraphrasing the information in a given
document. This requires the system to understand and reason about the text,
and to identify the most important or relevant information to include in the
answer. Abstractive QA is strictly related to applications such as
Summarization, where the goal is to generate a concise and coherent summary of
a document or set of documents.

Since Abstractive QA is a generative task such as Generative QA, the metrics
used for the evaluation are mostly the same: human evaluation when possible or
automatic evaluation with metrics like BLEU, ROUGE and BERT-Score otherwise.

##  Appendix B Datasets

###  B.1 GLUE

The GLUE (General Language Understanding Evaluation) benchmark is a collection
of diverse datasets designed to evaluate the performance of Natural Language
Understanding (NLU) models. The benchmark consists of nine individual
datasets, each targeting a different aspect of language understanding. We omit
the WNLI dataset in our evaluation as similar works [Devlin et al., 2019;
Clark et al., 2020] because even a trivial majority classifier performs better
than most of the models considered in this work. Here are detailed
descriptions of each dataset:

  * •

CoLA (Corpus of Linguistic Acceptability): This dataset focuses on grammatical
correctness. It contains sentences from a variety of sources, including books,
articles, and websites. The task is to determine whether a given sentence is
grammatically correct or not.

  * •

SST-2 (Stanford Sentiment Treebank): The SST-2 dataset is derived from movie
reviews, where each sentence is labeled with its sentiment (positive or
negative). The task is to classify the sentiment of each sentence accurately.

  * •

MRPC (Microsoft Research Paraphrase Corpus): This dataset comprises sentence
pairs extracted from various sources, such as online forums and news. The goal
is to determine whether the two sentences in each pair are paraphrases or not.

  * •

QQP (Quora Question Pairs): The QQP dataset is created from pairs of questions
asked on Quora, a question-and-answer platform. The task is to predict whether
two questions are semantically equivalent or not.

  * •

STS-B (Semantic Textual Similarity Benchmark): STS-B contains sentence pairs
along with their similarity scores. The dataset covers diverse domains such as
news, captions, and forum threads. The objective is to predict the degree of
semantic similarity between the two sentences.

  * •

MNLI (Multi-Genre Natural Language Inference): MNLI consists of sentence pairs
from various genres, such as fiction, government reports, and telephone
conversations. The goal is to determine the relationship between the two
sentences: entailment, contradiction, or neutral.

  * •

QNLI (Question-answering Natural Language Inference): QNLI is derived from the
Stanford Question Answering Dataset (SQuAD). It involves sentence pairs where
one sentence is a question and the other is a snippet from a Wikipedia
article. The task is to determine whether the given question can be answered
correctly using the provided snippet.

  * •

RTE (Recognizing Textual Entailment): The RTE dataset consists of sentence
pairs collected from news articles. The objective is to determine if the
premise sentence entails, contradicts, or remains neutral compared to the
hypothesis sentence.

  * •

WNLI (Winograd Natural Language Inference): WNLI focuses on resolving pronoun
references using contextual understanding. The dataset contains sentence pairs
with ambiguous pronouns, and the task is to determine if the pronoun in the
second sentence refers to the same entity as in the first sentence.

These datasets cover a wide range of Natural Language Understanding tasks,
including syntactic and semantic analysis, sentiment classification,
paraphrase identification, textual entailment, and pronoun resolution. They
serve as a standardized benchmark to evaluate the performance of different
language models and techniques in a variety of linguistic tasks.

##  Appendix C Alternative Efficient Objectives

###  C.1 RTS vs C-RTS

In this Section, we provide more details about the comparison between RTS and
C-RTS on a longer pre-training, which should better highlight the differences
between the two algorithms.

In Figure C.1 we show the results of the comparison between RTS and C-RTS over
the 4 datasets. We do not show the plot for ASNQ because fine-tuning the
models on this dataset every 10K steps is very expensive. We repeated every
fine-tuning on the downstream tasks 5 times with different initialization
seeds. On 3 benchmarks out of 4, C-RTS is superior thanks to the harder
training task. Choosing more challenging replacements instead of random ones
forces the model to find subtler relations between tokens. On QNLI, both
objectives perform equally from a statistical viewpoint. We argue that Small
models are more influenceable than Base counterparts because they have less
parameters. Thus, larger datasets with thousands of examples such as QNLI
reduce the differences seen while pre-training.

$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$\cdot 10^{5}$$70$$75$$80$Pre-
Training StepsWikiQA MAPRTSC-RTS

$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$\cdot 10^{5}$$80$$85$$90$Pre-
Training StepsTREC-QA MAPRTSC-RTS

$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$\cdot 10^{5}$$75$$80$$85$Pre-
Training StepsMRPC AccuracyRTSC-RTS

$0.2$$0.4$$0.6$$0.8$$1$$1.2$$1.4$$1.6$$1.8$$2$$\cdot 10^{5}$$84$$86$$88$Pre-
Training StepsQNLI AccuracyRTSC-RTS

Figure C.1: Performance comparison of RTS and C-RTS on WikiQA, TREC-QA, MRPC
and QNLI. We show both average and standard deviation after different fine-
tunings with 5 different random seeds.

###  C.2 Negative Results

In scientific research, negative results hold valuable lessons for future
investigations. This Section presents experiments and observations that
diverged from initial hypotheses, offering important insights to guide future
researchers.

####  C.2.1 C-RTS sampling always from the same Cluster

We conduct an experiment where we simplify C-RTS by replacing tokens only with
tokens within the same cluster. However, we discover that it is essential to
allow sampling from other clusters as well because the model demonstrates the
ability to learn how tokens are clustered after a significant number of
training steps.

####  C.2.2 Position-based techniques

In this experiment, we explore altering the positions of tokens by masking
certain positional encodings, similar to the MLM approach. The objective is to
determine the original position of masked tokens in the original text.
Although the classification head for this technique is smaller than MLM
(slightly larger than RTS due to BERT’s 512 possible positions), the results
on the GLUE benchmark are about 3.0 points worse.

We also define another objective where we (i) shuffle the positions of some
input tokens and (ii) predict their original positions. We randomly select 15%
of the tokens and permute them. Although this approach performs 1.5 points
below MLM on the GLUE average, it is almost as fast as RTS. Combining
position-based objectives with token-level objectives could be a potential
future research direction.

####  C.2.3 LM head-on ELECTRA’s discriminator

Based on the successful performance of SLM, we have implemented and tested a
version of SLM for ELECTRA. However, applying SLM directly to the ELECTRA
discriminator is not feasible because predictions are only made for altered
tokens, which undermines the task of detecting fake tokens. To address this,
we propose an alternative objective called SLM-All. In SLM-All, instead of
predicting only on tampered tokens, the model is tasked with predicting the
original values of all tokens while also reproducing the unchanged inputs in
the output.

We evaluated this approach on GLUE, WikiQA, ASNQ, and TREC-QA. Although it
required more time to be pre-trained (1.42 times the time required by a
similarly sized MLM-based model), the results were generally inferior compared
to other approaches.

The significant efficiency gap between SLM-All and other models is due to the
fact that the latter predicts MLM-like tokens for every output embedding of
each token, not just the 15% as in SLM or MLM. Even with a 25% reduction in
the number of training steps for the ELECTRA model (to balance the presence of
the additional generator with size 1/3), it still utilizes slightly more FLOPS
than BERT-MLM. Specifically, it achieved a GLUE average score of 80.02 on the
test leaderboard and MAP scores of 78.7, 86.7, 64.9 in WikiQA, TREC-QA, and
ASNQ, respectively.

##  Appendix D Self-Supervised Objectives for AS2

###  D.1 Negative Results

In this Section, we briefly describe the experiments that didn’t lead to
accuracy improvements of our language models. We believe negative results are
important to help other authors at selecting future research directions.

####  D.1.1 Additional positives from Large Language Model

Model | ASNQ |  | ASNQ + 1 Positive |  | WikiQA |  | WikiQA + 1 Positive
---|---|---|---|---|---|---|---
MAP | P@1 |  | MAP | P@1 |  | MAP | P@1 |  | MAP | P@1
RoBERTaBase [Liu et al., 2019b] | 66.9 (0.1) | 61.8 (0.1) |  | 67.1 (0.2) | 62.4 (0.2) |  | 85.8 (1.3) | 78.3 (2.8) |  | 85.7 (1.2) | 77.9 (1.9)
RoBERTaBase \+ SSP | 68.1 (0.2) | 64.1 (0.3) |  | 68.6 (0.2) | 64.3 (0.1) |  | 88.7 (0.3) | 82.9 (0.7) |  | 87.6 (1.1) | 80.9 (2.1)
RoBERTaBase \+ SP | 68.3 (0.1) | 64.1 (0.2) |  | 68.2 (0.3) | 63.6 (0.3) |  | 87.7 (0.3) | 81.0 (0.8) |  | 86.9 (0.2) | 79.2 (0.9)
RoBERTaBase \+ PSD | 67.7 (0.2) | 62.6 (0.4) |  | 67.1 (0.2) | 61.9 (0.3) |  | 86.4 (1.1) | 80.5 (1.6) |  | 86.8 (1.2) | 80.2 (2.3)
RoBERTaBase \+ ALL | 68.0 (0.1) | 63.9 (0.4) |  | 67.7 (0.1) | 63.8 (0.2) |  | 88.2 (0.4) | 82.5 (0.9) |  | 87.5 (0.9) | 81.1 (1.9)
ELECTRABase [Clark et al., 2020] | 67.5 (0.2) | 62.4 (0.4) |  | 68.1 (0.3) | 63.8 (0.4) |  | 85.0 (2.6) | 77.1 (4.0) |  | 82.5 (1.5) | 73.1 (1.7)
ELECTRABase \+ SSP | 69.7 (0.2) | 65.3 (0.3) |  | 67.6 (0.2) | 63.2 (0.2) |  | 88.6 (1.4) | 82.5 (2.0) |  | 87.1 (1.3) | 80.0 (2.6)
ELECTRABase \+ SP | 69.0 (0.1) | 65.0 (0.2) |  | 68.7 (0.3) | 64.0 (0.4) |  | 88.1 (1.5) | 81.8 (2.3) |  | 85.9 (0.6) | 77.8 (1.1)
ELECTRABase \+ PSD | 68.9 (0.3) | 65.3 (0.4) |  | 68.0 (0.2) | 62.3 (0.2) |  | 85.6 (0.7) | 78.6 (0.7) |  | 85.1 (1.2) | 77.5 (1.3)
ELECTRABase \+ ALL | 69.3 (0.2) | 65.0 (0.2) |  | 68.8 (0.3) | 64.3 (0.5) |  | 87.3 (1.2) | 80.8 (1.9) |  | 86.2 (1.4) | 78.5 (2.3)

Table D.1: Results in the data augmentation setting on ASNQ and WikiQA.

Most of the datasets for AS2 have a limited number of positive question-answer
pairs. For example, the training set of ASNQ has about 1.1 positive answers
every 356 candidates. The other datasets have instead a slightly higher ratio
between positive answer and the total number of candidates sentences for each
query: $\frac{0.5}{9.6}$ in WikiQA, $\frac{5.4}{46}$ in TREC-QA,
$\frac{0.23}{15}$ in WQA and about $\frac{1.7}{28}$ in NewsAS2 and TriviaAS2.
However, the gap between positives and negatives examples is still very large.

For those reasons, we try to increment the number of positive answer candidate
by exploiting Falcon-40B-instruct [Almazrouei et al., 2023], a Large Language
Model with 40 billion parameters fine-tuned for instruction-based
conversations. Specifically, we generate a single additional positive answer
for each query by providing Falcon with the question and possibly by adding a
question mark at the end if it was missing.

Notice that we do not add positive answer candidates to the development and
test sets to perform a fair comparison with the results provided in the
previous Sections. We test the vanilla RoBERTaBase and ELECTRABase checkpoints
as well as the models further pre-trained with our specialized objectives on
ASNQ and WikiQA. The latter should provide a meaningful test suite being a
very large dataset with very few positives and a small dataset with a
reasonable number of positive answers, respectively.

We report the results of the experiment in Table D.1. On ASNQ, there are rare
combinations of models and objectives that benefits from the additional
positives. For example, the two baselines RoBERTaBase and ELECTRABase show an
improvements of the P@1 by 0.6 points and 1.2 points respectively. Our
continuously pre-trained models generally perform lower by 1.0 P@1 points when
fine-tuned with the additional positives, apart from RoBERTaBase \+ SSP, which
gains 0.2 points in P@1.

On WikiQA, no model benefits from the additional positives. In particular,
there is a decrease between 0.3 and 4.0 points in P@1, depending on the
combination between models architectures and specialized our pre-training
objectives.