# Classification of Infant Sleep/Wake States: Cross-Attention among Large
Scale Pretrained Transformer Networks using Audio, ECG, and IMU Data

Kai Chieh Chang12, Mark Hasegawa-Johnson12, Nancy L. McElwain24, Bashima
Islam3 1 Department of Electrical and Computer Engineering, University of
Illinois at Urbana Champaign, USA 2 Beckman Institute for Advanced Science and
Technology, University of Illinois at Urbana-Champaign, USA 4 Department of
Human Development and Family Studies, University of Illinois at Urbana-
Champaign, USA 3 Department of Electrical and Computer Engineering, Worcester
Polytechnic Institute, USA
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Infant sleep is critical to brain and behavioral development. Prior studies on
infant sleep/wake classification have been largely limited to reliance on
expensive and burdensome polysomnography (PSG) tests in the laboratory or
wearable devices that collect single-modality data. To facilitate data
collection and accuracy of detection, we aimed to advance this field of study
by using a multi-modal wearable device, LittleBeats (LB), to collect audio,
electrocardiogram (ECG), and inertial measurement unit (IMU) data among a
cohort of 28 infants. We employed a 3-branch (audio/ECG/IMU) large scale
transformer-based neural network (NN) to demonstrate the potential of such
multi-modal data. We pretrained each branch independently with its respective
modality, then finetuned the model by fusing the pretrained transformer layers
with cross-attention. We show that multi-modal data significantly improves
sleep/wake classification (accuracy = 0.880), compared with use of a single
modality (accuracy = 0.732). Our approach to multi-modal mid-level fusion may
be adaptable to a diverse range of architectures and tasks, expanding future
directions of infant behavioral research.

##  1 Introduction

Sleep is a physiological and behavioral process central to brain development
during infancy [1, 2, 3, 4]. During the first six months of life, infants
spend more time sleeping than awake [5]. Further, indices of infant sleep
quality and quantity are associated with subsequent cognitive and language
development [6, 7, 8, 9, 10, 11], attention regulation [12, 13, 14], social-
emotional functioning [10, 12], and physical health [15, 16]. Given the
importance of infant sleep to development, automated and unobtrusive
monitoring of sleep is critical. Thus, we focus on the task of sleep/wake
classification in this study.

Figure 1: LittleBeats (center) is capable of collecting audio, ECG, and IMU
data. Reproduced with permission from [17]

Laboratory-based polysomnography (PSG) is the gold-standard sleep assessment
for adults and infants alike [18, 19]. However, given the high level of burden
and expense of laboratory PSG, researchers have used wearable devices for
sleep monitoring [20]. Notably, past studies of sleep/wake classification
using wearable devices typically extract sleep features from single-modality
data. For example, with actigraphy data, [21] carried out extensive studies on
adult sleep/wake classification using traditional signal processing methods,
classic machine learning methods, and several deep learning methods.
Electrocardiogram (ECG) signals are also often used because stable QRS complex
and longer R peak period can indicate sleep stages [22]. Studies such as [23]
used ECG data to classify adult sleep/wake with a 5-layer convolutional neural
network (CNN) concatenated with 2 fully connected (FC) layers. A signal-
processing-based approach was suggested by [24], where audio recordings of
adults’ respiratory sounds were used to estimate sleep/wake likelihood from
signal level features such as autocorrelation. All published studies of
automatic sleep/wake classification use different datasets, therefore their
accuracies are not strictly comparable, but the accuracies reported in all
four of these studies are between 81% and 84% (see Table 1 for details). A few
studies have used bimodal data to further improve classification performance.
Ref. [25] concatenated the frequency bins from respiratory signals and ECG to
feed into a FC based network, achieving classification accuracy of 85.3%. Ref.
[26] uses random forest to classify sleep/wake from acceleration and heart
rate data collected by Apple Watch, achieving 87.3% accuracy. Recent work tend
to focus on a more involved sleep classification with 5 stages. Ref. [27]
extracted infant vocalizations from audio recordings and physical motions from
video recordings, and classified sleep stages using a random forest, resulting
in an average accuracy near 85%. Supratak et al.’s DeepSleepNet [28] and their
subsequent stage of the art (SOTA) TinySleepNet [29] used only
electroencephalogram (EEG) to reach 87.5% in accuracy.

To achieve further improvements in accuracy, we leveraged trimodal data
collected by an infant wearable device, LittleBeats (LB), as shown in Fig. 1.
LB can synchronously record infant sounds (via microphone [audio]), physiology
(via 3-lead ECG), and motion (via inertial measurement unit [IMU]) for
extended periods of time (8-10 hours) in the home context.

To perform sleep/wake classification using LB multi-modal data, we developed
an ensemble of three large scale transformer networks pretrained on unimodal
audio, ECG, and IMU datasets, and fine-tuned using trimodal data. This system
has the potential to combine all important feature extraction done by the
research mentioned above, including the extraction of vocalization and
breathing patterns from audio recording, QRS complex features from ECG, and
position and motion from IMU. To the best of our knowledge, our study is the
first to access these three modalities jointly on sleep/wake classification.
We show superior classification performance using all three modalities
compared with any single modality or pairs of modalities combined.

In contributing to neural network (NN) architectures, we explore the benefits
of pretraining large scale transformer networks on unlabeled audio, ECG, and
IMU data, then finetuning a cross-attention based fusion architecture on a
small LB labeled dataset. Unsupervised pretraining involves training a NN on a
large, diverse dataset, which enables it to learn generic latent features that
can be applied to a wide range of tasks. There is a wide range of pretraining
schemes for audio, ECG, and IMU data. Specifically, wav2vec 2.0 (W2V2) [30]
showed the benefits of pretraining audio using contrastive loss, achieving
state of the art performance in tasks such as automatic speech recognition.
Ref. [31] combined W2V2 and contrastive multi-segment coding to pretrain on
ECG signals, achieving outstanding performance on cardiac arrhythmia
classification and patient identification. LIMU-BERT [32] is derived from
bidirectional encoder representations from transformers (BERT) [33], and is
used to pretrain transformers with unlabeled IMU data. Inspired by the above
architectures, we pretrained three branches of large scale transformer
networks, one for each modality, and were able to extract latent features
useful for sleep/wake classification.

To utilize features from all three modalities, we implemented a fusion
strategy that relies on cross-attention among pretrained transformer layers.
Multimodal fusion is often done by early fusion, where the input from each
modality is concatenated and fed into the network as a single input, or late
fusion, where each modality is processed separately and the outputs are
combined at the final stage. We define early fusion as fusing right after
feature extraction and not fusion of raw data. Much work has been done on
these 2 fusion techniques. Ref. [34] employed early fusion by cascading
learned features from ECG and IMU directly, and passed them through a dense
network. Ref. [35] described a simple late fusion of different branches by
summing or averaging the logit outputs. Recently, intermediate fusion has
become more common. This type of feature fusion happens in the middle of the
NN, and uses a variety of architectures depending on the tasks. Ref. [36]
extensively used so-called “cross-stitch” modules for soft parameter fusion,
taking multiple linear weighted sums of each feature extraction layer as input
for the next layer. Cross-attention based fusion techniques were explored in
recent papers such as [37] [38], where the attention layers take concatenated
features from different modalities as input, and [39], where a single cross-
attention layer is used to share information between branches. Our approach is
innovative, first, in that it relies on the pretrained transformer layers from
each branch, rather than training a feature sharing mechanism from scratch.
Second, we fuse the three transformer networks by alternating self and cross-
attention at different layers, which both preserves each branch’s transformer
features and incorporates attention from other modalities.

In sum, in addition to being the first study to combine audio, ECG, and IMU
for classification of infant sleep/wake states, we develop an innovative
cross-attention based fusion for large scale pretrained transformer networks
to combine three modalities. We not only reiterate the well known benefit of
pretraining, but also demonstrate the ability of pretrained unimodal
transformers to be fine-tuned using cross-attention-based multi-modal fusion
to improve accuracy. We believe our work lays the groundwork for improved
accuracy in a wide variety of signal processing tasks by the use of multimodal
wearable devices such as LB.

##  2 Method

Figure 2: The LittleBeats Model Architecture. Each of the audio (top), ECG
(center), and IMU (bottom) branches consists of feature extraction
(convolutional and/or feedforward projection), positional encoding, and 12
transformer layers. Each branch is pretrained individually on unlabeled data.
During labeled finetuning, the three branches are combined via cross-attention
at the feature level. Their outputs are then concatenated and fed into dense
layers to output logits.

Our architecture is an ensemble of three similarly structured transformer
networks, with each branch targeting audio, ECG, or IMU data respectively, as
shown in Fig. 2.

For the audio branch, we utilize a standard 12-layer W2V2 [30] network because
W2V2 repeatedly shows superb performance in various speech-related tasks such
as automatic speech recognition and phoneme recognition. A recent study [40]
showed an oracle W2V2 pretrained on 4300 hours of unlabeled infant-family
audio data collected by LB or the Language ENvironment Analysis (LENA) system
[41] increased performance of speech diarization and vocalization type
classifications. As infant vocalization features may well include cues of
sleep, we adopt pretrained weights from [40] for the audio W2V2 branch.

As for ECG data, we were inspired by the similarity between speech and ECG. As
early as 1940, researchers [42] were actively using impulse trains to imitate
glottal excitation for speech generation. R-peaks in ECG data have a similar
time domain structure, so a speech-centric NN should be able to learn ECG
features. Speech and ECG differ in one respect: speech contains periodicity at
many frequencies (pitch and formants), while ECG is dominated by the inter-
beat interval, therefore ECG features may be learnable using less pre-training
than speech. We pretrain a standard 12-layer W2V2 using 574 hours of unlabeled
ECG data to utilize our observations above and to match the structure of the
audio branch.

IMU data does not share similar structure as audio or ECG: an IMU signal is
composed of parallel 3-axis accelerometer and 3-axis gyroscope signals, whose
correlations are important for signal interpretation. To take full advantage
of the information contained in our IMU data, we used a pretraining paradigm
based on LIMU-BERT [32]. In place of W2V2’s CNN feature extraction layers,
LIMU-BERT begins with a single fully-connected feature projection layer per
input sample, followed by several layers of transformers. This network is
pretrained with Masked Language Model objectives and a Span Masking mechanism.
We pretrained 574 hours of IMU data collected from LB, on a modified LIMU-BERT
architecture where we lifted the restriction that all transformers share
weights, extended the number of layers to 12, and matched the transformer
implementation to the one found in W2V2.

Figure 3: Cross-attention for the audio branch. Multiheaded attention layer
(MHA) takes the audio embedding as query. IMU and ECG outputs from the other
two branches first get linearly projected and concatenated into a 2-channel
embedding. The embedding is reduced to 1-channel by a 1-d convolution layer
and passed into MHA as key and value.

To maximize the model’s ability to learn across modalities, or to extend the
transformer attention beyond a single modality, we utilized linear
transformation and cross-attention every 4 transformer layers, as inspired by
[43] and their work using cross-attention across audio channels. An example of
the cross-attention mechanism for the audio branch is shown in Fig. 3. A
cross-attention layer for the audio branch consists of multiple cross-
attention heads, each of which takes its query from the previous layer’s audio
embedding, and its values and keys from a fusion of the previous layer’s ECG
and IMU branches. ECG and IMU embeddings are first passed through two
different linear layer projections producing an output with the frame rate and
embedding dimension of the audio, then concatenated to create a 2 channel
embedding. A channel-wise 1d convolutional layer is then used to create the
vector that serves as key and value of the audio cross-attention layer. The
multiple audio cross-attention heads are then concatenated, added to a
residual connection (from the previous audio layer), normalized, and passed
through a feedforward layer. The ECG and IMU cross-attention layers have the
same architecture, with each modality receiving query inputs from its own
previous layer, while each modality’s key and value inputs come from the other
two modalities.

Mathematically, we can denote each hidden input embedding to the transformer
layer as $\mathbf{H}_{k}\in\mathbb{R}^{N\times L_{e}}$, where the indices
$(k,l,m)$ are any permutation of $\\{\text{audio},\text{ECG},\text{IMU}\\}$,
$N$ is the number of samples in the input, and $L_{e}$ is the architecture
embedding size. Let’s define the query, key, and value projection of each
branch to be $\mathbf{W}^{\\{Q,K,V\\}}\in\mathbb{R}^{L_{e}\times L_{e}}$.
Let’s constrain the discussion of attention regarding one head of one branch
where $k=\text{audio}$. The attention layers can be self attention or cross-
attention depending on the layer index $i$. We found using cross-attention at
layers 1, 5, and 9 (i.e. $i\%4=1$ for layers $0\leq i\leq 11$) gives the best
performance.

$\displaystyle\text{SelfAttn}_{i,k}=\text{softmax}\left(\frac{\mathbf{H}_{k}\mathbf{W}_{i}^{Q}\left(\mathbf{H}_{k}\mathbf{W}_{i}^{K}\right)^{T}}{\sqrt{L_{e}}}\right)\mathbf{H}_{k}\mathbf{W}_{i}^{V}$

$\displaystyle\text{CrossAttn}_{i,k}=\text{softmax}\left(\frac{\mathbf{H}_{k}\mathbf{W}_{i}^{Q}\left(\mathbf{H}_{\text{cross}}\mathbf{W}_{i}^{K}\right)^{T}}{\sqrt{L_{e}}}\right)\mathbf{H}_{\text{cross}}\mathbf{W}_{i}^{V}$

$\displaystyle\mathbf{H}_{\text{cross}}=\mathbf{A}\left[\begin{matrix}\text{proj}(\mathbf{H}_{l}^{T})\\\
\text{proj}(\mathbf{H}_{m}^{T})\end{matrix}\right]$

where $\mathbf{A}\in\mathbb{R}^{L_{e}\times 2}$ is a trainable mixing weight.
This discussion is applicable to the other heads and other branches.

The three branches’ outputs are concatenated and passed through three dense
layers with ReLU activations to produce binary logits for sleep and wake. With
the pretrained weights loaded onto each branch and the ensemble in place, we
use frame-wise cross entropy loss to finetune the network.

##  3 Experiment

###  3.1 Dataset

Participants were recruited via study flyers distributed to local community
organizations as well as online listservs serving families of young children.
In the context of larger studies of child behavioral development, study
coordinators provided participating families with a LB or LENA kit, along with
instructions for conducting associate recordings in the home. Families were
asked to complete 2 to 3 daylong (8+ hours) recordings over a 2-week period.
All study procedures were approved by the Institutional Review Board at the
University of Illinois Urbana-Champaign.

Unlabeled data included (a) 4300hr of audio home recordings (1000hr collected
by LB and 3200hr by LENA) from 245 families with children under 5 years of age
(see [40] for further details), as well as (b) 574hr of ECG data and (c) 574hr
of IMU data collected by LB from 28 families with children (54% female) under
5 years of age (Mean = 26 months, Range: 3-65 months). For the labeled data,
we gathered 68.5hr of synchronized audio, ECG, and IMU data from the same 28
families. Four trained human annotators labeled infant sleep and wake states
from 1.5hr segments of LB audio files. Annotation of sleep states was
facilitated by referring to (a) parental reports of their child’s activities
while wearing the LB device in the home, (b) visual inspection of the wav form
of the audio file, and (c) listening to audio recordings for indicators of
infant sleep (e.g., slowed steady breathing, no vocalizations or movement).

We randomly divide the labeled data into training set (50hr, 25 families),
validation set (4.5hr, 1 family), and testing set (14hr, 2 families). Data
from families/infants in the training, validation, and testing datasets did
not overlap.

Figure 4: Zeroing data according to timestamp. For each data segment, the
recorded UTC time multiplied by sampling frequency
($F_{s,k}*(\mathbf{T}_{k}^{\text{end}}[i]-\mathbf{T}_{k}^{\text{start}}[i])$)
is longer than the recorded number of samples
($\mathbf{S}_{k}^{\text{end}}[i]-\mathbf{S}_{k}^{\text{start}}[i]$). The
missing data are simply filled with zeros.

###  3.2 Data Preprocessing

As a first step, we synchronized data across modalities. LB records each
modality in slightly different time segments. For each segment, LB records the
timestamps in absolute UTC start time and end time, and the associated start
and end sample index. Formally, let’s define

$\displaystyle\mathbf{T}_{k}^{\text{start}},\mathbf{T}_{k}^{\text{end}},\mathbf{S}_{k}^{\text{start}},\mathbf{S}_{k}^{\text{end}}\in\mathbb{R}^{L_{k}}$

where for each modality, each element in $\mathbf{T}$ is the absolute UTC
start or end time, each element in $\mathbf{S}$ is starting or ending data
sample index, $L$ is the number of data chunks.

LB is designed to function continuously despite variability in the number of
nonzero IMU samples, therefore it sometimes fails to record all samples in a
given audio or ECG segment. Such missed samples can be detected by comparing
the known ground truth duration of a segment,
$F_{s,k}*(\mathbf{T}_{k}^{\text{end}}[i]-\mathbf{T}_{k}^{\text{start}}[i])$,
to the number of samples it contains,
$(\mathbf{S}_{k}^{\text{end}}[i]-\mathbf{S}_{k}^{\text{start}}[i])$, where
$F_{s,k}$ is the sampling frequency for modality $k$. When an incomplete
segment is detected, we fill in the missing data with zeros. We obtain
$\mathbf{\hat{Z}}_{k}$ where each of the vectors has length
$F_{s,k}*(\mathbf{T}_{k}^{\text{end}}[i]-\mathbf{T}_{k}^{\text{start}}[i])$.
This is shown in Fig. 4.

Timestamps of different modalities are approximately but not precisely
aligned, because the writing to SD card on LB is asynchronous. In order to
make multimodal processing possible, we simply discard the data that did not
overlap according to the timestamp and truncate $\mathbf{\hat{Z}}_{k}$ to
$\mathbf{Z}_{k}\in\mathbb{R}^{L}$ where L is the same across all modalities.
$\mathbf{Z}_{k}$ is then segmented into 30s segments $\mathbf{X}_{k}$.

Human annotators mark the beginning and ending of each period of sleep. These
times are used to assign, to each 30s segment $\mathbf{X}_{k}$, a binary label
$\mathbf{Y}_{k}\in\\{0,1\\}$ where 0 is wake and 1 is sleep. If both sleep and
wake labels are present in segment $k$, $\mathbf{Y}_{k}$ is set equal to the
label with longer duration.

###  3.3 Metric

Because this is a binary classification task, we evaluated standard accuracy,
precision, recall, F1-score, and Cohen’s kappa $\kappa$.

Table 1: Results Comparing Architectures with Different Number and Types of Modalities. Results in the first section are our work; results in the second section are baselines computed using the same test set; results in the third section are copied from the corresponding reference papers, and were therefore computed using incomparable test sets. Test Set | Architecture | Modality |  $\uparrow$ Accuracy |  $\uparrow$ Precision |  $\uparrow$ Recall | $\uparrow\text{F}1$ |  $\uparrow$ Kappa
---|---|---|---|---|---|---|---
LB 68.5 hr | W2V2-mod | Audio | 0.797 | 0.794 | 0.909 | 0.848 | 0.546
| W2V2-mod | ECG | 0.732 | 0.704 | 0.980 | 0.819 | 0.347
| LIMU-BERT-mod | IMU | 0.786 | 0.837 | 0.815 | 0.826 | 0.549
| Proposed | Audio + ECG + IMU | 0.880 | 0.881 | 0.934 | 0.907 | 0.741
LB 68.5 hr | FFT + Dense [25] mod | Audio + ECG | 0.444 | 0.729 | 0.168 | 0.273 | 0.0523
| CNN [23] mod | ECG | 0.565 | 0.494 | 0.434 | 0.634 | 0.0991
| Random Forest [26] | ECG + IMU | 0.717 | 0.729 | 0.799 | 0.762 | 0.410
| CNN + LSTM [29] | ECG | 0.843 | 0.825 | 0.778 | 0.748 | 0.639
Adult 7.1 hr | Signal Processing [24] | Audio | 0.833 | - | 0.922 | - | 0.508
Adult 37k hr | LSTM [21] | Actigraphy | 0.831 | 0.816 | 0.914 | 0.855 | -
Adult 292 hr | FFT + Dense [25] | Respiratory + ECG | 0.853 | - | - | - | -
Adult 110 hr | CNN [23] | ECG | 0.815 | 0.478 | 0.930 | 0.360 | 0.260
Adult 244 hr | Random Forest [26] | ECG + IMU | 0.873 | - | 0.895 | - | 0.396

###  3.4 Implementation Details

Audio sample rate is downsampled from 24000Hz to 16000Hz and ECG sample rate
is upsampled from 2381Hz to 16000Hz, to match the default sampling rate of
W2V2. IMU data are kept at a sampling rate of 150Hz because upsampling to
16000Hz might introduce too many artifacts. We did not experiment with data
interpolation to fill out the zeros during time synchronization nor with data
augmentation to diversify the data.

For audio and ECG W2V2 branch, we used almost the same baseline structure as
described in [30], with 12 transformer layers, hidden size 768, intermediete
size 3072 and 16 attention heads. Convolution feature extraction structure
remain the same with hidden size 512, kernel size [10,3,3,3,3,2,2] and strides
[5,2,2,2,2,2,2]. For IMU branch, the feature extraction is done by a linear
layer, a normalization layer, and adding a position embedding. The decoding
part has 12 transformer layers, hidden size 72, intermediate size 144 and 4
attention heads. Outputs of the three branches are concatenated into a vector
of size 1608, passed through a dense structure as follows: a linear layer from
size 1608 to size 1608, ReLU, linear layer from size 1608 to 804, dropout of
rate 0.1, linear layer from 804 to 2.

Each experiment was run for 2 epochs with batch size 16, with the standard
Adam optimizer. This was done on a single RTX A6000 GPU. Training code was
based on Huggingface Transformers and the code is available for inspection and
further development. 111This footnote will be replaced by a github URL
containing an open-source implementation upon acceptance of the manuscript.

###  3.5 Baseline Implementation

The majority of sleep/wake classification baselines did not release code, so
we implemented some of their algorithms with modification based on their
description in the paper. Specifically, for [25] we replaced the respiratory
input with LB audio data, and trained with gradient descent instead of
Levenberg–Marquardt back-propagation algorithm. For [23], we used increasing
strides ([1, 2, 3, 4, 5, 6]) for the CNN layers instead of using [2, 2, 2, 2,
2, 2] because each LB ECG segments has more samples than theirs. We also
modified the dropout value to 0.1 based on empirical testing. [26] used 3-axis
acceleration and heart rate data collected on Apple Watch as their dataset. We
calculated heart rate at each recorded segment from LB ECG data, using the
python package HeartPy. We then input our timestamp, acceleration data, and
calculated heart rate to their released code. Finally, state of the art sleep
stage classifier [29] released their code online. We used ECG as input to the
model instead of EEG in the original paper.

##  4 Results

###  4.1 Benefits of Three Modalities

The first section in Table 1 shows how using all modalities together compares
to using each of the modalities alone for the purpose of sleep/wake
classification. The single modality architecture takes each branch in Fig. 2,
with pretrained weights, and adds the final three FC layers for
classification. The proposed architecture is the entire model pretrained on
unlabeled data and finetuned with proposed cross-attention at every 4 layers.
We see that, by quite a large margin, using all three modalities together as
proposed, the network is able to learn more about infants’ sleep patterns and
achieve better performance across the board than using just a single modality.
Note that while recall for the ECG branch outperforms the proposed network by
5%, its precision is lower by 20%, suggesting that the unimodal ECG network is
a worse classifier since it too frequently classifies segments as “sleep.”

The second section in Table 1 presents results of key baseline implementation
on the LB dataset, as described in Section 3.5. Because [25] and [23] have not
released their code and [26] and [29] are not designed to work strictly on ECG
data, results need to be interpreted with caution. However, we can still
observe that our proposed model using all three modalities outperforms all key
baselines, which only use one or two modalities.

Numbers in the last section in Table 1 are results reported in other papers,
using other datasets, and are therefore not strictly comparable to the numbers
in the first two sections. The classifiers summarized by these results are
sleep/wake classifiers using audio only [24], ECG only [23], ECG and
respiratory signals [25], motion [21], and ECG and IMU [26]. While all
datasets are different and comparison of the accuracies is therefore not
theoretically justifiable, such a comparison nevertheless supports the
conclusion displayed in the top half of the table, viz., that sleep/wake
classification performed using three modalities is more accurate than
sleep/wake classification performed using only one or two modalities.

###  4.2 Benefits of Pretraining

Table 2 shows how pretraining affects the performance of the proposed
architecture. Here we configured the model to use all three branches, but only
load the pretrained weights for specific modalities. Cross-attention is
applied at every $4^{\text{th}}$ layer for the 12 layers of transformers. We
see that every time we add pretrained weights for a particular branch, we have
a performance increase across all metrics. This reinforces our understanding
of pretrained networks. By pretraining each branch using the unlabeled data,
then fine-tuning the entire system together against a small number of labeled
data, we are able to get better performance.

###  4.3 Fusion Techniques

Table 2: Results when Pretraining Different Branches of the Architecture Pretrained |  $\uparrow$ Accuracy |  $\uparrow$ Precision |  $\uparrow$ Recall | $\uparrow\text{F}1$ |  $\uparrow$ Kappa
---|---|---|---|---|---
None | 0.797 | 0.829 | 0.850 | 0.839 | 0.565
IMU | 0.837 | 0.874 | 0.863 | 0.868 | 0.655
Audio+IMU | 0.871 | 0.858 | 0.949 | 0.901 | 0.715
ECG+IMU | 0.875 | 0.866 | 0.944 | 0.904 | 0.726
All | 0.880 | 0.881 | 0.934 | 0.907 | 0.741
Table 3: Results of Early, Late, and the Proposed Fusion Technique Fusion |  $\uparrow$ Accuracy |  $\uparrow$ Precision |  $\uparrow$ Recall | $\uparrow\text{F}1$ |  $\uparrow$ Kappa
---|---|---|---|---|---
Early | 0.779 | 0.857 | 0.773 | 0.813 | 0.544
Late | 0.798 | 0.781 | 0.947 | 0.856 | 0.530
Proposed | 0.880 | 0.881 | 0.934 | 0.907 | 0.741

Our discussion is incomplete without comparison to early and late fusion
techniques discussed in the literature [35]. Table 3 presents performance of 2
fusion methods against the proposed cross-attention-based intermediate fusion
technique. Early fusion is inspired by [34], where convolutional features
extracted from ECG and IMU data are concatenated with anthropometric data as
input to a downstream neural network. In our variation, we concatenate the
three outputs from the feature extractor for each branch as shown in Fig. 2,
skipping the transformer layers, and pass through the dense network for fine-
tuning. As for late fusion, we leave in the transformer layers, triplicate the
FC layers to generate logits for each modality, and average the three logits
for evaluation. The proposed cross-attention fusion achieved better
performance in almost every metric except that it has a lower recall than late
fusion. Similar to the trend found in Table 1, this high recall for late
fusion results from a disproportional confusion matrix, as shown by late
fusion’s low precision score. Therefore, the proposed fusion architecture is
still favorable in general.

###  4.4 Ablation Study

Table 4 shows the effect of using cross-attention at different layers. All
models were fine-tuned on labeled multimodal data after loading pretrained
weights for all three modalities separately. “-” means the architecture did
not converge. We empirically found that cross-modal attention once every 4
layers ($i\%4=1$) gives the best performance across the board. Many models
with cross-modal attention fail to converge: convergence failure occurs with
cross-modal attention every second layer, and with cross-modal attention in
the middle four layers ($4\leq i<8$) or the last four layers ($8\leq i<12$).
Cross-modal attention in the first four layers ($0\leq i<4$) converged to a
system with accuracy lower than that of the system with no cross-modal
attention. The sensitivity of cross-modal attention to architectural
configuration was an unexpected result. We defer this to future studies.

Table 4: Result with Cross-Attention Applied at Various Layers. The equation in the first column specifies the layers in which cross-modal attention is applied, e.g., $i\%4=1$ means that cross-modal attention was performed in layers 1, 5, and 9, where layers are numbered $0\leq i\leq 11$. Cross |  $\uparrow$ Accuracy |  $\uparrow$ Precision |  $\uparrow$ Recall | $\uparrow F1$ |  $\uparrow$ Kappa
---|---|---|---|---|---
None | 0.765 | 0.788 | 0.851 | 0.818 | 0.487
i%2=1 | - | - | - | - | -
i%4=1 | 0.880 | 0.881 | 0.934 | 0.907 | 0.741
i%6=1 | 0.836 | 0.870 | 0.865 | 0.868 | 0.652
$0\leq i<4$ | 0.755 | 0.811 | 0.791 | 0.801 | 0.484
$4\leq i<8$ | - | - | - | - | -
$8\leq i<12$ | - | - | - | - | -

##  5 Conclusion

With the development of multi-modal wearable devices, we are able to gather
synchronized audio, ECG, and IMU data for the task of infant sleep/wake
classification. We demonstrated the best classification performance when using
all three modalities compared with our own single modality network or
single/double modality network found in the literature. In addition, we
developed an ensemble of large scale pretrained transformer neural network, by
fusing the pretrained transformer layers with cross-attention at every 4
layers. This fusion method is not limited to the task of wake/sleep
classification, but seems likely to generalize successfully to any multi-modal
network in which all modalities have the same number of pretrained transformer
layers. Our work presents exciting directions for multi-modal studies of
infant and child development.

## Acknowledgment

We would like to thank the families who participated in this research, as well
as Jordan Bodway and Jenny Baldwin for their assistance with data collection
and processing. This work was supported by funding from the National Institute
of Mental Health (R21MH112578), National Institute of Drug Abuse
(R34DA050256), the National Institute of Food and Agriculture (ILLU-793-368),
and the Personalized Nutrition Initiative and Center for Social and Behavioral
Science at the University of Illinois at Urbana-Champaign through the Seed
Grant program.

## References

  * [1] A. R. Tarullo, P. D. Balsam, and W. P. Fifer, “Sleep and infant learning,” _Infant Child Dev._ , vol. 20, no. 1, pp. 35–46, Jan. 2011.
  * [2] M. S. Blumberg, A. J. Gall, and W. D. Todd, “The development of sleep-wake rhythms and the search for elemental circuits in the infant brain,” _Behav. Neurosci._ , vol. 128, no. 3, pp. 250–263, Jun. 2014.
  * [3] R. E. Dahl, “The regulation of sleep and arousal: Development and psychopathology,” _Dev Psychopathol_ , vol. 8, no. 3, pp. 3–27, 1996.
  * [4] E. Bathory and S. Tomopoulos, “Sleep regulation, physiology and development, sleep duration and patterns, and sleep hygiene in infants, toddlers, and preschool-age children,” _Curr. Probl. Pediatr. Adolesc. Health Care_ , vol. 47, no. 2, pp. 29–42, Feb. 2017.
  * [5] B. C. Galland, B. J. Taylor, D. E. Elder, and P. Herbison, “Normal sleep patterns in infants and children: a systematic review of observational studies,” _Sleep medicine reviews_ , vol. 16, no. 3, p. 213—222, June 2012\.
  * [6] A. Hupbach, R. L. Gomez, R. R. Bootzin, and L. Nadel, “Nap-dependent learning in infants,” _Dev. Sci._ , vol. 12, no. 6, pp. 1007–1012, Nov. 2009.
  * [7] K. Horváth, K. Myers, R. Foster, and K. Plunkett, “Napping facilitates word learning in early lexical development,” _J. Sleep Res._ , vol. 24, no. 5, pp. 503–509, Oct. 2015.
  * [8] E. Dearing, K. McCartney, N. L. Marshall, and R. M. Warner, “Parental reports of children’s sleep and wakefulness: longitudinal associations with cognitive and language outcomes,” _Infant Behav. Dev._ , vol. 24, no. 2, pp. 151–170, Feb. 2001.
  * [9] A. Bernier, S. M. Carlson, S. Bordeleau, and J. Carrier, “Relations between physiological and cognitive regulatory systems: infant sleep regulation and subsequent executive functioning,” _Child Dev._ , vol. 81, no. 6, pp. 1739–1752, Nov. 2010.
  * [10] M. Ednick, A. P. Cohen, G. L. McPhail, D. Beebe, N. Simakajornboon, and R. S. Amin, “A review of the effects of sleep during the first year of life on cognitive, psychomotor, and temperament development,” _Sleep_ , vol. 32, no. 11, pp. 1449–1458, Nov. 2009.
  * [11] A. Scher, “Infant sleep at 10 months of age as a window to cognitive development,” _Early Hum. Dev._ , vol. 81, no. 3, pp. 289–292, Mar. 2005\.
  * [12] A. Sadeh, G. De Marcas, Y. Guri, A. Berger, L. Tikotzky, and Y. Bar-Haim, “Infant sleep predicts attention regulation and behavior problems at 3-4 years of age,” _Dev. Neuropsychol._ , vol. 40, no. 3, pp. 122–137, 2015\.
  * [13] M. Thunström, “Severe sleep problems in infancy associated with subsequent development of attention-deficit/hyperactivity disorder at 5.5 years of age,” _Acta Paediatr._ , vol. 91, no. 5, pp. 584–592, 2002.
  * [14] F. V. O’Callaghan, A. Al Mamun, M. O’Callaghan, A. Clavarino, G. M. Williams, W. Bor, H. Heussler, and J. M. Najman, “The link between sleep problems in infancy and early childhood and attention problems at 5 and 14 years: Evidence from a birth cohort study,” _Early Hum. Dev._ , vol. 86, no. 7, pp. 419–424, Jul. 2010.
  * [15] J.-P. Chaput, C. E. Gray, V. J. Poitras, V. Carson, R. Gruber, C. S. Birken, J. E. MacLean, S. Aubert, M. Sampson, and M. S. Tremblay, “Systematic review of the relationships between sleep duration and health indicators in the early years (0-4 years),” _BMC Public Health_ , vol. 17, no. Suppl 5, p. 855, Nov. 2017.
  * [16] L. Matricciani, C. Paquet, B. Galland, M. Short, and T. Olds, “Children’s sleep and health: A meta-review,” _Sleep Med. Rev._ , vol. 46, pp. 136–150, Aug. 2019.
  * [17] “About littlebeats.” [Online]. Available: https://littlebeats.hdfs.illinois.edu/about-littlebeats/
  * [18] A. Sadeh, “Iii. sleep assessment methods,” _Monogr. Soc. Res. Child Dev._ , vol. 80, no. 1, pp. 33–48, Mar. 2015.
  * [19] S. E. Beck and C. L. Marcus, “Pediatric polysomnography,” _Sleep Med. Clin._ , vol. 4, no. 3, pp. 393–406, Sep. 2009.
  * [20] A. Sadeh, “Iii. sleep assessment methods,” _Monogr. Soc. Res. Child Dev._ , vol. 80, no. 1, pp. 33–48, Mar. 2015.
  * [21] J. Palotti, R. Mall, M. Aupetit, M. Rueschman, M. Singh, A. Sathyanarayana, S. Taheri, and L. Fernandez-Luque, “Benchmark on a large cohort for sleep-wake classification with machine learning techniques,” Jun 2019.
  * [22] C. Cajochen, J. Pischke, D. Aeschbach, and A. A. Borbély, “Heart rate dynamics during human sleep,” _Physiology & Behavior_, vol. 55, no. 4, pp. 769–774, 1994.
  * [23] J. Malik, Y.-L. Lo, and H. tieng Wu, “Sleep-wake classification via quantifying heart rate variability by convolutional neural network,” _Physiological Measurement_ , vol. 39, no. 8, p. 085004, aug 2018.
  * [24] E. Dafna, A. Tarasiuk, and Y. Zigel, “Sleep-wake evaluation from whole-night non-contact audio recordings of breathing sounds,” p. e0117382, Feb 2015.
  * [25] W. Karlen, C. Mattiussi, and D. Floreano, “Sleep and wake classification with ecg and respiratory effort signals,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 3, no. 2, pp. 71–78, 2009.
  * [26] O. Walch, Y. Huang, D. Forger, and C. Goldstein, “Sleep stage prediction with raw acceleration and photoplethysmography heart rate data derived from a consumer wearable device,” _Sleep_ , vol. 42, no. 12, 08 2019, zsz180.
  * [27] S. Cabon, F. Porée, A. Simon, B. Met-Montot, P. Pladys, O. Rosec, N. Nardi, and G. Carrault, “Audio- and video-based estimation of the sleep stages of newborns in neonatal intensive care unit,” _Biomedical Signal Processing and Control_ , vol. 52, pp. 362–370, 2019.
  * [28] A. Supratak, H. Dong, C. Wu, and Y. Guo, “DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 25, no. 11, pp. 1998–2008, nov 2017.
  * [29] A. Supratak and Y. Guo, “Tinysleepnet: An efficient deep learning model for sleep stage scoring based on raw single-channel eeg,” _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, pp. 641–644, 2020.
  * [30] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
  * [31] J. Oh, H. Chung, J. myoung Kwon, D. gyun Hong, and E. Choi, “Lead-agnostic self-supervised learning for local and global representations of electrocardiogram,” 2022.
  * [32] H. Xu, P. Zhou, R. Tan, M. Li, and G. Shen, “Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications,” in _Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems_ , ser. SenSys ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 220–233.
  * [33] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  * [34] Z. Ni, T. Wu, T. Wang, F. Sun, and Y. Li, “Deep multi-branch two-stage regression network for accurate energy expenditure estimation with ecg and imu data,” p. 3224–3233, Oct 2022.
  * [35] K. Gadzicki, R. Khamsehashari, and C. Zetzsche, “Early vs late fusion in multimodal convolutional neural networks,” in _2020 IEEE 23rd International Conference on Information Fusion (FUSION)_ , 2020, pp. 1–6.
  * [36] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, “An improved event-independent network for polyphonic sound event localization and detection,” 2021.
  * [37] L. Ying, H. Yu, J. Wang, Y. Ji, and S. Qian, “Multi-level multi-modal cross-attention network for fake news detection,” _IEEE Access_ , vol. 9, pp. 132 363–132 373, 2021.
  * [38] X. Wei, T. Zhang, Y. Li, Y. Zhang, and F. Wu, “Multi-modality cross attention network for image and sentence matching,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 10 938–10 947.
  * [39] X. Song, H. Chao, X. Xu, H. Guo, S. Xu, B. Turkbey, B. J. Wood, T. Sanford, G. Wang, and P. Yan, “Cross-modal attention for multi-modal image registration,” _Medical Image Analysis_ , vol. 82, p. 102612, 2022.
  * [40] J. Li, M. Hasegawa-Johnson, and N. L. McElwain, “Towards robust family-infant audio analysis based on unsupervised pretraining of wav2vec 2.0 on large-scale unlabeled family audio,” 2023.
  * [41] D. Xu, J. A. Richards, and J. Gilkerson, “Automated analysis of child phonetic production using naturalistic recordings,” _J. Speech Lang. Hear. Res._ , vol. 57, no. 5, pp. 1638–1650, Oct. 2014.
  * [42] H. Dudley, “The vocoder—electrical re-creation of speech,” _Journal of the Society of Motion Picture Engineers_ , vol. 34, no. 3, pp. 272–278, 1940.
  * [43] F.-J. Chang, M. Radfar, A. Mouchtaris, B. King, and S. Kunzmann, “End-to-end multi-channel transformer for speech recognition,” 2021.


# Reasoning about Ambiguous Definite Descriptions

Stefan F. Schouten  Peter Bloem  Ilia Markov  Piek Vossen
Vrije Universiteit Amsterdam
<EMAIL_ADDRESS>
###### Abstract

Natural language reasoning plays an increasingly important role in improving
language models’ ability to solve complex language understanding tasks. An
interesting use case for reasoning is the resolution of context-dependent
ambiguity. But no resources exist to evaluate how well Large Language Models
can use explicit reasoning to resolve ambiguity in language. We propose to use
ambiguous definite descriptions for this purpose and create and publish the
first benchmark dataset consisting of such phrases. Our method includes all
information required to resolve the ambiguity in the prompt, which means a
model does not require anything but reasoning to do well. We find this to be a
challenging task for recent LLMs. Code and data available at:
https://github.com/sfschouten/exploiting-ambiguity

##  1 Introduction

Natural language understanding and reasoning are interdependent skills:
reasoning with natural language presupposes a level of understanding; but full
understanding may require the resolution of ambiguities through reasoning.
Complex ambiguity in particular could benefit from explicit ‘out loud’
reasoning, such as the reasoning that is produced with chain-of-thought
prompts.

Existing resources used to evaluate reasoning are not well suited to
investigate the capability of resolving ambiguity by explicit reasoning. Some
existing benchmarks require the resolution of ambiguity, but focus only on
ambiguity that humans can resolve intuitively (e.g. Winograd schemas, Levesque
et al. 2012). The ability to reason with natural language is often evaluated
with tasks considered complex enough to require it. These may or may not
include ambiguities of various types, making them poorly suited to evaluate
when models are able to resolve ambiguity and which types. Such tasks may also
benefit from abilities besides explicit reasoning, such as factual recall.
Thus, improvements on these tasks cannot be easily attributed to improvements
in reasoning.

Figure 1: Example ambiguous definite description. Since a person’s native
language does not change over time, we know that the de dicto interpretation
is correct.

In this paper we create a new benchmark dataset which requires models to
resolve ambiguous definite descriptions. Definite descriptions are phrases
that denote entities by describing the properties or roles that are unique to
them within the relevant context (e.g. “the pope”, “john’s mother”, “our
king”). We use ambiguous descriptions which denote one of two entities, and
include information on both entities in context. Specifically, we introduce a
de dicto / de re ambiguity by including a temporal operator (see Figure 1 for
an example). By asserting something that is true of only one of these
entities, one of the two interpretations can be excluded by reasoning.

We demonstrate the value of this approach by creating a new benchmark dataset
generated from Wikidata. We explicitly include the knowledge required for
disambiguation in the prompt of each instance. Doing so ensures that when a
system answers incorrectly, its failure stemmed from its inability to
adequately reflect on the given information. We perform experiments showing
the performance of recent large language models on our benchmark. We find
that: (1) GPT4 and chain-of-thought prompting perform best (2) the LLaMA-based
OpenAssistant model beats GPT3.5 in average accuracy, although the latter is
more consistent; and (3) de re instances are harder than de dicto instances
for all models.

In summary, our contributions are  (a) an outline of ambiguous definite
descriptions, an under-explored class of problems useful for the evaluation of
systems that reason with and about natural language (section 3)  (b) a new
benchmark dataset consisting of ambiguous definite descriptions, showcasing
the value of the problem class (section 4)  (c) experimental results showing
the performance of recent large language models on this benchmark (section 5)

##  2 Related Work

Reasoning is a broad term and the tasks related to it are varied. We focus on
the type of reasoning that is explicit and is done with natural language,
which has recently been surveyed extensively (Qiao et al., 2023; Huang and
Chang, 2023; Yu et al., 2023). This paradigm became possible with the advent
of large language models (LLMs) such as ChatGPT and GPT4 OpenAI (2023). These
models can be prompted to perform ‘chain-of-thought’ reasoning Wei et al.
(2022); even in a zero-shot setting Suzgun et al. (2022). This has
significantly improved performance for many tasks Suzgun et al. (2022). Using
a new dataset of ambiguous definite descriptions, our experiments evaluate to
what extent LLMs can use (chain-of-thought) reasoning to resolve ambiguity in
language.

Recent work on ambiguity includes the construction of both curated Liu et al.
(2023); Min et al. (2020) and synthetic datasets Yuan et al. (2023). Such
datasets investigate ambiguity in a variety of tasks such as: natural language
inference, open-domain question answering, etc. Our dataset is synthetically
generated (but factual), and focuses specifically on de dicto / de re
ambiguities, forming a binary classification task.

The term ‘logical reasoning over natural language’ (LRNL) was coined by Yang
et al. (2023) to talk about a new trend where natural language is used to
represent knowledge and LLMs are used to reason over that knowledge. One
challenge this paradigm could have to overcome is the potential for ambiguity
inherent in natural language. Our work intentionally introduces ambiguity, and
evaluates how well LLMs can resolve it by reasoning.

Previously, Winograd schemas Levesque et al. (2012) have been used in various
benchmarks to test for commonsense reasoning. These schemas consist of
sentences with ambiguous pronouns which must be resolved in one of two ways.
However, existing schemas focus on ambiguities that humans resolve intuitively
(without the need for explicit reasoning). Creating new schemas that do
require explicit reasoning does not appear straightforward, especially since
creating them has proved difficult in general Kocijan et al. (2023). Our
method also involves ambiguous noun phrases, but requires explicit reasoning
about definite descriptions rather than implicit reasoning about pronouns.

##  3 Ambiguous Definite Descriptions

Definite descriptions are noun phrases that denote entities by describing the
properties or roles that are unique to them within the relevant context.
Examples of such phrases in English include among others: “the pope”, “john’s
mother”, and “our king”.

### De dicto vs. de re

The kind of ambiguities we use are known as de dicto / de re ambiguities. They
involve a statement that is either true of what is said (de dicto) or true of
the thing (de re). Take for example the sentence depicted in Figure 1, where
it is unclear if the description “the pope” denotes the current pope (Francis)
or the previous pope (Benedict). The source of the ambiguity is the phrase “In
2010”, which can be read as primarily relating to the description (“the
pope”), meaning the property is ascribed to the pope from 2010 (Benedict). Or,
it can be read as relating primarily to the property ascription itself (“was a
native speaker of German”), in which case the property is being ascribed to
the current pope but is qualified as being true in 2010.

This type of ambiguity is the result of descriptions being combined with a
non-extensional operator of which the temporal ‘In PERIOD’ is just one
example. Other non-extensional operators could be used, such as modal
operators, or propositional attitudes like ‘PERSON believes that’ or ‘PERSON
hopes that’.

| Component |  Explanation |  Example |
---|---|---|---|---
Premises |  entity -relations |  A number of relations of the entity and the context in which it is true. |  Lars Lervik was part of the Telemark Battalion from January 2010 to January 2013. |
Lars Lervik was part of the Brigade Nord from January 2018 to January 2020. |
relation -properties |  The value of the property for each relation. |  Brigade Nord is a military unit of size class brigade. |
Telemark Battalion is a military unit of size class battalion. |
regularity |  The relevant rule or regularity that is required to resolve the ambiguity. |  Military units do not change size class. |
Question | context |  Establishes the current context. |  If the current date is June 2019, |
sentence |  An ambiguous sentence with a temporal operator and a property being ascribed to an entity denoted by a definite description. |  what is the most likely interpretation of the following sentence: The military unit of Lars Lervik was of size class brigade in March 2010. |
interpretations |  Two interpretations for the ambiguous sentence, one of which contradicts the regularity. |  1\. Lars Lervik’s unit in March 2010 (Telemark Battalion) was of size class brigade |
2\. Lars Lervik’s current unit (Brigade Nord) was of size class brigade in March 2010 |
Table 1: An example from the RADD-Wikidata-5-EN dataset (property pair
P7779_Q21506450). The parts of the example that are bold are the same for each
instance. For this example instance the de re interpretation is correct.

##  4 RADD-Wikidata-5-EN

To demonstrate the value of Reasoning about Ambiguous Definite Descriptions we
create a (semi-)automatically generated dataset based on Wikidata. This
dataset is based on 5 Wikidata property-pairs, and is in English. We focus on
creating ambiguous sentences with a temporal operator. Wikidata contains many
triples qualified with the ‘start time’ and ‘end time’ properties to indicate
the period in which they were true.

In Table 1, one can see a complete example instance from this dataset. For a
given main entity (e.g., Lars Lervik) we find the relations corresponding to
that entity (Lars Lervik’s military units and the beginning and end of his
term with them). This information is combined with a property of the relations
that does not change over time (the size class of the military units). Pairs
of these relations are sampled such that the property (the size class) has
different values for the two relations (in this case, one brigade and one
battalion). Note that, by changing the property we ascribe to the denotee, we
can flip which interpretation is correct (battalion instead of brigade) so we
can generate two mirror instances for each entity and relation-pair.

We prioritized finding a small diverse set of property-pairs. Besides
requiring a property whose value changes over time and a property whose value
does not, the combination of properties also must yield sufficient results on
wikidata (we required at least 50 to create the 100 samples). We also filtered
out property-pairs where the results were dominated by a single entity, for
example, had the data for the ‘person-military_unit-size_class’ pair consisted
of only a handful of people changing units many different times, we would not
have included it. See Table 2 for details on which property pairs we include
and what they represent.

We include all facts relevant to the main entity in each prompt whether they
are necessary to resolve the ambiguity or not, this allows us to measure if
models can work around distractor facts.

| property-pair | description | #instances | #premises / prompt
---|---|---|---|---
| P6_P19 | head of government birth place | 7299 | 41.14
| P286_P17 | head coach country | 1607 | 17.50
| P159_P206 | headquarter location in or next to body of water | 147 | 12.03
| P7779_Q21506450 | military unit size class | 90 | 8.21
| P8047_P30 | ship country of registry continent | 74 | 6.37
Table 2: Property-pairs with their marks in Figure 2 and descriptions, the
number of instances (each yielding a de dicto and de re prompt), and average
number of premises per prompt.

###  4.1 Template design considerations

We use templates to generate the instances in the dataset, and while
experimenting we identified a few things to avoid when phrasing them.

Avoiding disambiguation by verb tense. The ambiguous description must not
contain a verb whose tense favours one interpretation over the other. For
example with: “In MONTHYEAR, the club for which X is/was head coach was from
Y.” If we use ‘X is head coach’ it implies that ‘In MONTHYEAR’ does not scope
over the definite description, whereas with ‘X was head coach’ the implication
is that it does scope over the definite description (assuming MONTHYEAR is in
the past). Thus, we avoid using such phrases, for example, by formulating the
example as: “In MONTHYEAR, the club with X as its head coach was from Y.”

Avoiding event-referencing properties. We also avoid referencing events in the
property ascription, which can cause ‘in MONTHYEAR’ to be read as referring to
the time of the event, for example: “In MONTHYEAR, the club with X as its head
coach was founded by Y.” Sometimes we can avoid this by rephrasing: “In
MONTHYEAR, the club with X as its head coach had Y as its founder.” One
property-pair was discarded because we could not find a good way to phrase the
sentence, this was about the location of formation (P740) of distributors of
creative works (P750).

##  5 Experiments

We evaluate three dialogue-oriented large language models (LLMs) on our
benchmark in a zero-shot setup. The models we include are

  1. (a)

Koala-13B† Geng et al. (2023)111hf.co/TheBloke/koala-13B-GPTQ-4bit-128g

  2. (b)

LLaMA-33B† Touvron et al. (2023) fine-tuned on version 7 of OpenAssistant
Conversations Köpf et al. (2023)222hf.co/TheBloke/OpenAssistant-
SFT-7-Llama-30B-GPTQ

  3. (c)

GPT-3.5(-turbo-0301) OpenAI (2022)

  4. (d)

GPT-4(-0613) OpenAI (2022)

Models marked with $\dagger$ were used with 4-bit OPTQ quantization Frantar et
al. (2023).

The evaluation is performed on 500 instances (100 per property-pair). To
prompt the language models, we start with instances such as the one from the
‘example’ column in Table 1 and number the premises P$1$ through P$N$. Next,
we shuffle the order of the interpretations, and then append one of the
following two instructions: direct=“Answer only with "Option 1" or "Option 2",
explain your decision after.” or chain-of-thought=“When answering, let’s
consider both options, and think step by step. DO NOT repeat the premises,
only refer to their number.” When using chain-of-thought (CoT), we follow up
with a second message: “Based on this, what is your final answer, "Option 1"
or "Option 2"?” The models were prone to repeating the premises in their
answer, thus warning them not to do this was necessary to avoid them from
running out of context-window before providing the final answer.

Koala-13BOAsst-33BGPT3.5GPT4$.4$$.6$$.8$AccuracyDirectCoTP286_P17P6_P19P8047_P30P159_P206P7779_Q21506450 Figure 2: Accuracies obtained by each model on the RADD-Wikidata-5-EN benchmark. Bars show average accuracy, marks overlaid on top indicate accuracy on each of the five property-pairs. |  | de dicto |  | de re
---|---|---|---|---
|  | Pr | Re | F1 |  | Pr | Re | F1
90Direct | Koala-13B | 0.46 | 0.42 | 0.44 |  | 0.47 | 0.47 | 0.47
OAsst-33B | 0.59 | 0.79 | 0.67 |  | 0.61 | 0.36 | 0.45
GPT3.5 | 0.53 | 1.00 | 0.69 |  | 0.97 | 0.10 | 0.18
GPT4 | 0.56 | 1.00 | 0.72 |  | 1.00 | 0.20 | 0.33
90CoT | Koala-13B | 0.56 | 0.51 | 0.53 |  | 0.56 | 0.54 | 0.54
OAsst-33B | 0.65 | 0.84 | 0.73 |  | 0.80 | 0.51 | 0.61
GPT3.5 | 0.64 | 0.82 | 0.72 |  | 0.73 | 0.48 | 0.56
GPT4 | 0.76 | 0.99 | 0.86 |  | 0.99 | 0.67 | 0.79
Table 3: Precision, recall and F1 score by class (columns), for each prompt styles and model (rows). Bold and italic show best scores overall and per prompt-style respectively. |  | Direct |  | CoT |
---|---|---|---|---|---
|  | 55de dicto | 55de re | 55neither |  | 55de dicto | 55de re | 55neither |
Koala-13B | de dicto | 21.2 | 27.2 | 1.6 |  | 25.4 | 20.6 | 4.0 |
de re | 25.0 | 23.6 | 1.4 |  | 19.8 | 26.8 | 3.4 |
OAsst-33B | de dicto | 39.6 | 6.2 | 4.2 |  | 42.2 | 6.0 | 1.8 |
de re | 27.6 | 18.0 | 4.4 |  | 22.8 | 25.4 | 1.8 |
GPT3.5 | de dicto | 49.8 | 0.2 | 0.0 |  | 41.0 | 8.8 | 0.2 |
de re | 45.0 | 5.0 | 0.0 |  | 23.8 | 23.8 | 2.4 |
GPT4 | de dicto | 50.0 | 0.0 | 0.0 |  | 49.6 | 0.4 | 0.0 |
de re | 39.8 | 10.2 | 0.0 |  | 16.2 | 33.6 | 0.2 |
Table 4: Average confusion. The columns show prompt styles and predicted
classes, the rows show models and label classes. Correct predictions are in
gray.

###  5.1 Quantitative results

Figure 2 shows the accuracies obtained by each model for each property-pair
and on average. The precision, recall and F1 scores for each model, class, and
prompt style can be seen in Table 3. The highest average accuracy is obtained
by GPT4 (83.2%), followed by OAsst-33B (67.7%) and GPT3.5’s (64.8%). The mean
accuracy of Koala-13B is roughly equal to random guessing (52.2%). Looking at
the performance for the various property-pairs, we can see that GPT3.5,
although less accurate overall, is more consistent than OAsst-33B. The
performance for OAsst-33B is particularly low on the P6_P19 pair. This can be
explained by the average number of premises (41.14) which is over twice the
amount as included for the other property pairs (6.37 - 17.5 on average, never
fewer than 5). The long prompts exceed the number of tokens included in the
context window during training for OAsst-33B (and Koala-13B). Note also that
each model benefits from the use of chain-of-thought prompting.

In Table 4, we can see the confusion table for each model and prompt style. We
can see that there are (up to 4%) cases where the models do not produce a
parseable result. Finally, we observe that the de re instances are the more
difficult class; only GPT4 with chain-of-thought is able to perform better
than random guessing on these instances.

###  5.2 Error Analysis

We perform a small analysis of the answers given by the best-performing model
GPT4 to give insight into what areas need the most improvement.

About half of the mistakes made by GPT4 involve the model not properly
following the chain-of-thought instruction, meaning the model already made a
prediction in the first one or two sentences of its response. Although,
ignoring this instruction seems to be common among both the de dicto and de re
classes. It seems that absent any reasoning the models have a strong
preference for predicting de dicto (this can also be seen from the confusion
matrix in Table 4). This explains why this behaviour mostly produces errors on
the de re class.

About a third of all de re chain-of-thought answers conclude that neither
option is correct. Although despite of this GPT4 almost always still predicts
one of the two options. This seems to indicate that the models completely fail
to consider the de re option. Take for example the following ambiguous
sentence “The birth place of Prague 6’s head of state was Třebíč in November
2001" for which the correct interpretation is Option 2: ‘Prague 6’s current
head of state (Marie Kousalíková) was, as of November 2001, born in Třebíč’.
GPT4 reasons as follows: “Option 2 can be eliminated based on premises (P6)
and (P9), as Marie Kousalíková was born in Třebíč, but she was not the head of
state in November 2001, according to (P1) and (P2). Hence, this interpretation
would be incorrect.” But, this reasoning is clearly incorrect, since her being
head of state in November 2001 is not required for Option 2 to be correct.
This type of error seems to be a large part of why the models perform worse
for de re instances.

Together these two types of errors appear to make up the vast majority of
GPT4’s mistakes.

##  6 Conclusion

We have introduced Reasoning about Ambiguous Definite Descriptions as a way to
evaluate how well systems can use natural language reasoning to resolve
ambiguous language and have created the first benchmark dataset to demonstrate
the value of this task. Our findings show that recent LLMs are not yet capable
of solving this task reliably, and that they particularly struggle with de re
instances. Our error analysis suggests that models—besides not always
‘thinking step-by-step’ as instructed—do not properly consider the de re
options, instead hallucinating extra conditions for their correctness.

In future work, we will expand our work by: (1) testing how changing the
provided information (e.g. removing regularities) affects the performance; (2)
embedding this problem in other tasks such that the ability to solve it
depends on the resolution of the ambiguity; (3) constructing multi-lingual
versions to evaluate if the ability to resolve ambiguity through reasoning is
independent of the language; and (4) extending our method to other extensional
operators, and other types of ambiguities.

## Limitations

The de dicto / de re ambiguity we use in our dataset is one of many possible
kinds of ambiguities. Completely excluding the possibility that our results
rely on the particulars of this ambiguity will require a diverse set of
ambiguities.

So far we have only included English prompts in our benchmark. We leave the
creation of prompt templates in other languages for future work.

We have performed a ‘best-effort’ tuning of the prompts. It is plausible that
with a more extensive tuning better prompts can be found and that overall
performance could be improved to some degree. An extensive tuning may also
reveal that each model benefits from different prompts, which could also
change their relative performance.

## Acknowledgements

This research was supported by Huawei Finland through the DreamsLab project.
All content represented the opinions of the authors, which were not
necessarily shared or endorsed by their respective employers and/ or sponsors.

## References

  * Frantar et al. (2023) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023.  OPTQ: Accurate Quantization for Generative Pre-trained Transformers.
  * Geng et al. (2023) Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.  Koala: A dialogue model for academic research.  Blog post.
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Towards Reasoning in Large Language Models: A Survey.  ArXiv:2212.10403 [cs].
  * Kocijan et al. (2023) Vid Kocijan, Ernest Davis, Thomas Lukasiewicz, Gary Marcus, and Leora Morgenstern. 2023.  The Defeat of the Winograd Schema Challenge.  ArXiv:2201.02387 [cs].
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.  OpenAssistant Conversations – Democratizing Large Language Model Alignment.  ArXiv:2304.07327 [cs].
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The Winograd Schema Challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_.
  * Liu et al. (2023) Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023.  We’re Afraid Language Models Aren’t Modeling Ambiguity.  ArXiv:2304.14399 [cs].
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering Ambiguous Open-domain Questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5783–5797, Online. Association for Computational Linguistics.
  * OpenAI (2022) OpenAI. 2022.  Introducing ChatGPT.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  ArXiv:2303.08774 [cs].
  * Qiao et al. (2023) Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023.  Reasoning with Language Model Prompting: A Survey.  ArXiv:2212.09597 [cs].
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022.  Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.  ArXiv:2210.09261 [cs].
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  ArXiv:2302.13971 [cs].
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of Thought Prompting Elicits Reasoning in Large Language Models.  _arXiv:2201.11903 [cs]_.  ArXiv: 2201.11903.
  * Yang et al. (2023) Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023.  Logical Reasoning over Natural Language as Knowledge Representation: A Survey.  ArXiv:2303.12023 [cs].
  * Yu et al. (2023) Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2023.  Natural Language Reasoning, A Survey.  ArXiv:2303.14725 [cs].
  * Yuan et al. (2023) Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar. 2023.  AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1023–1030, Dubrovnik, Croatia. Association for Computational Linguistics.

inline,caption=]TODO:

  * •

Must

  * •

Should

  * •

Could

    * E)

Make distractor’s optional.

    * E)

Test effect of removing certain premises, the regularity in particular.

    * P)

Correlation between number of distractors and model performance?

    * E)

OTHER LANGUAGE, Dutch would be easy to do.

  * •

Would

    * –