# Intelligent Sight and Sound:
A Chronic Cancer Pain Dataset

Catherine Ordun 1, 2, Alexandra N. Cha 1, Edward Raff 1, 2,
Byron Gaskin 1, Alex Hanson 1, Mason Rule 3, Sanjay Purushotham 2, James L.
Gulley 3
1 Booz Allen Hamilton
2University of Maryland, Baltimore County
3Center for Cancer Research, National Cancer Institute, National Institutes of
Health

###### Abstract

Cancer patients experience high rates of chronic pain throughout the treatment
process. Assessing pain for this patient population is a vital component of
psychological and functional well-being, as it can cause a rapid deterioration
of quality of life. Existing work in facial pain detection often have
deficiencies in labeling or methodology that prevent them from being
clinically relevant. This paper introduces the first chronic cancer pain
dataset, collected as part of the Intelligent Sight and Sound (ISS) clinical
trial, guided by clinicians to help ensure that model findings yield
clinically relevant results. The data collected to date consists of 29
patients, 509 smartphone videos, 189,999 frames, and self-reported affective
and activity pain scores adopted from the Brief Pain Inventory (BPI). Using
static images and multi-modal data to predict self-reported pain levels, early
models show significant gaps between current methods available to predict pain
today, with room for improvement. Due to the especially sensitive nature of
the inherent Personally Identifiable Information (PII) of facial images, the
dataset will be released under the guidance and control of the National
Institutes of Health (NIH).

##  1 Introduction

The prevalence of chronic pain in cancer patients is high, with an estimated
prevalence of 59% in those undergoing anticancer treatment, 64% of whom have
advanced stage disease and 33% who continue to experience pain following
completion of curative treatment [1]. Despite advances in pain management,
prompt assessment and management of cancer pain remains a challenge and a
large proportion of patients continue to experience moderate to severe pain.
Sub-optimal pain management can block patient recovery and improvement, making
the already difficult cancer experience, worse, for both patient and family
[2, 3]. Manual clinical assessment requires accounting for a landscape of
complex emotions and beliefs that clinicians must regularly take into account
when assessing cancer patient pain - physical, psychological, social, and
spiritual elements combined with severe distress for future outlook [4, 2].
For example, patients undergoing chemotherapy are more likely to believe that
“good patients" do not complain about pain which they believe can be
distracting to clinicians and become non-communicative [2]. Further, few
patients are actually screened for pain at each clinical visit [3], and pain
is under-reported in patient populations such as nursing home patients [3].
Due to the variety of complex conditions affecting cancer pain, experts
recommend repeated, regular pain assessment, which can be difficult and
impractical for manual assessment by clinicians.

Currently, no facial pain datasets exists for chronic cancer pain and little
research overall has been conducted into machine learning for the
identification and evaluation of chronic pain. For example, in a review by
[4], only seven out of 52 machine learning papers evaluated pain in a non-
acute context such as chronic fatigue, fibromyalgia, chronic pancreatitis,
migraines, and genetic pain. Existing facial pain research focuses on acute,
musculoskeletal pain such as chronic lower back pain [5] and shoulder pain [6,
7] or simulated pain induced by heat or electrical stimuli [8, 9] where
painful expressions are obvious through grimaces and eye raises. Such datasets
are manually labeled by trained observers with Facial Action Coding Units
[10], making the labeling procedure prohibitively expensive and impractical
for clinical use. Further, external pain labels may be biased towards an
outside observer’s impression of a patient’s pain, not the patient themselves.
Research also shows that typical pain facial expressions that correlate with
physical pain are less frequently observed among chronic pain cancer patients
who exhibit subdued and placid expressions [11].

Given the limitations of existing facial expression pain data, the U.S.
National Institutes of Health (NIH) National Cancer Institute (NCI) initiated
_“A Feasibility Study Investigating the Use of Machine Learning to Analyze
Facial Imaging, Voice and Spoken Language for the Capture and Classification
of Cancer Pain_ " [12], or "Intelligent Sight and Sound" (ISS). Details of the
protocol are available publicly at
https://clinicaltrials.gov/ct2/show/NCT04442425. This is an observational,
non-interventional clinical study that aims to address the following problem
statement [13, 12]: “To determine if a new observational based pain prediction
algorithm can be produced that is accurate to standard, patient-reported pain
measures and is generalizable for a diverse set of individuals, across sexes
and skin types.” The study has two objectives: 1) investigate facial image
data, and 2) analyze text and audio, as modalities for predicting self-
reported chronic cancer pain.

The study is ongoing and aims to recruit 112 patients. We report the initial
dataset, which is less than a quarter of the final data consisting of 29
patients. Data include multimodal extracts from video submitted in a
spontaneous, home setting, and in a few cases of in-clinic capture at the NIH.
It includes visual spectrum (RGB) video frames, facial images resulting from
face detection models, facial landmarks from Active Appearance Models (AAMs)
[14, 15], audio files, Mel spectrograms, audio features, and self-reported
pain scores adopted from the Brief Pain Inventory (BPI) [16, 17, 18].We will
present details of the study design, data distribution, and storage procedures
to ensure patient privacy. We also provide initial baseline results for pain
classification using simple, traditional, machine learning models and neural
networks.

##  2 Related Works

Automatically detecting pain from facial expressions has been extensively
published following methods of facial emotion recognition (FER). The majority
of these works have focused on acute or musculoskeletal physical pain [4, 19,
20, 21, 22, 23, 24, 25, 26, 27]. Primary pain datasets based upon facial
imaging include UNBC-McMaster Shoulder Pain Expression Archive [6, 7], the
Biopotential and Video Heat Pain (BioVid) Database using controlled, simulated
heat to induce pain [8], Multimodal Intensity Pain (MIntPAIN) database using
pain resulting from electrical stimulation [9], the Experimentally Induced
Thermal and Electrical (X-ITE) Pain Database [28, 29], and the EmoPain for
chronic, musculoskeletal pain [30]. These datasets traditionally contain video
sequences since video enables continuous clinical monitoring of pain response
[18]. These datasets also contain extensive offline annotations of pain
ratings by external observers, and sometimes include additional modalities
such as thermal and depth data. Additional video facial expression pain
datasets exist that focus on different patient populations, but primarily
focus on physical pain. These include multimodal behavioral and physiological
data for neonatal pain [31, 32] and the University of Regina (UofR) Pain in
Severe Dementia dataset [33, 34]. A summary of the pain datasets is provided
in Table 1.

Table 1: Related Pain Datasets.

Dataset Stimulus Subjects Frames Sequences Seq. Duration Modality UNBC-
McMaster [6, 7] Shoulder pain 25 48,398 200 10 - 30 sec., per Unimodal BioVid
[8] Heat stimulus 90 8700 87 5.5 sec. Multimoda MIntPAIN [9] Electronic
stimulus 20 187,939 9366 1 - 10 sec. Multimodal EmoPain [30] Chronic lower
back pain 22 44,820 35 3 sec. Multimodal Neonatal Pain, USF [35, 32, 36] Heel
lancing 31 3026 200 9 sec. Multimodal UofR [33] Physical, painful movements
102 162,629 95 Unknown Multimodal X-ITE [37, 28, 29] Heat and electronic
stimuli 134 26,454 N/A 7 sec. Multimodal ISS (Dec. 2020 - Jul. 2021) Chronic
cancer pain 29 189,999 509 3.52 - 135.59 Multimodal

##  3 ISS Dataset

The ISS protocol is a single site study with a goal of enrolling a total of
112 patients (90 adult and 22 pediatric) who are actively receiving treatment
for advanced malignancies and/or tumors at the NIH Clinical Center or treated
with standard of care in the community. The study is overseen by the NIH
Institutional Review Board (IRB), and the protocol was also reviewed by NCI’s
Center for Cancer Research (CCR) Scientific Review Committee. New patient
enrollment was paused during the Covid-19 pandemic due to initially unknown
risks, but has resumed with vaccine availability and clinician guidance.

###  3.1 Sample and Study Design

To obtain as representative a sample as possible within the constraints of a
feasibility study with an overall small sample size, the sample consists of
twelve cohort groups of seven patients each. Patients represent a breadth of
age, sex, skin tone (as a proxy for ethnicity), and pain experience. The
current ISS dataset consists of 29 adult patients ages 18 years and over who
have consented to participate in the study; no pediatric patients (ages 12-17)
have been enrolled yet.

Table 2: ISS: Twelve Patient Cohorts.

Number Pain Target Skin Types Sex Pain Class Goal Current 1A 0 I - III Male
None 7 7 1B 0 I - III Female None 7 2 1C 0 IV - VI Male None 7 4 1D 0 IV - VI
Female None 7 0 2A 1-3 I - III Male Low 7 1 2B 1-3 I - III Female Low 7 1 2C
1-3 IV - VI Male Low 7 3 2D 1-3 IV - VI Female Low 7 0 3A 4-6 I - III Male
Moderate 7 2 3B 4-6 I - III Female Moderate 7 2 3C 4-6 IV - VI Male Moderate 7
0 3D 4-6 IV - VI Female Moderate 7 0 4A 7-10 I - III Male Severe 7 1 4B 7-10 I
- III Female Severe 7 2 4C 7-10 IV - VI Male Severe 7 2 4D 7-10 IV - VI Female
Severe 7 3 112 30

The goal is to evenly split the sample by i) sex (Male or Female), ii)
Fitzpatrick Skin Type [38], a self-reported, visual method of skin tone
classification, where patients are asked to type themselves into one of two
groups: “light" skin tones in types I-III or “dark" skin tones in types IV-VI,
and iii) a self-reported “worst" pain score reported on a 0 – 10 Numerical
Rating Scale (NRS) [39]. The self-reported pain score is referred to as the
“Pain Target" and are grouped into levels 0, 1-3, 4-6, and 7-10.

It represents the worst pain the patient has experienced in the past thirty
days prior to the start of the study. It is fixed throughout the patient’s
enrollment and does not change. As a result, there is no variance for the
“Pain Target" score. The “Pain Target" is the classification target which is
later used for our baseline tasks. The twelve different cohorts are shown in
Table 2, along with the goal of seven patients to be enrolled per cohort, and
the current distribution of patients enrolled.

Table 3: ISS Study Design Overview. * Note that due to the global Covid-19
pandemic, the majority of patient videos were submitted in the remote setting.

Study Duration 3 months # Remote Submission 3 / week; Max. 1 /day # In-Clinic
Submissions 1 - 4 /week* Survey Tool Smartphone (iPhone or Android) or
computer (camera/mic.) Time / Submission Average total time: 3 min Time /
Question Q 1-9: 1 min, Q10: 15 sec, Q11: 15 sec. - 3 min. Self-Reported Pain Q
1-9: Questions with Likert-scale responses Voice/Video: Prompt Q10: Read and
record one of 3 randomized nursery rhymes. Voice/Video: Narrative Q11: Record
respond to "Describe how you feel right now." Compensation: Min. 3 / week,
they earn $15.

Note that data from one patient (0009) in Cohort 2B was unusable. As a result
our analysis reports across 29 patients. Clinical inclusion criteria include
individuals with a diagnosis of a cancer or tumor who are under active
treatment for this condition at NIH/NCI. Patients must also have access to a
smart phone or computer with camera, microphone, and internet access. Several
clinical exclusion criteria apply. Excluded are patients with active central
nervous system (CNS) metastases, with the exception of those who have
completed curative intent radiotherapy or surgery and have been asymptomatic
for three months prior to consent, patients with Parkinson’s disease, and any
psychiatric condition that would prohibit the understanding or rendering of
informed consent. Additional exclusion criteria those who are non-English
speaking or have known current alcohol or drug abuse. Each patient is enrolled
for a three-month period and are financially incentivized to complete three
check-ins per week remotely and up to four in-clinic check-ins. The study
design is summarized in Table 3. Patients engage using an electronic
questionnaire and through video recording using a custom developed mobile or
web application, using an Android, iPhone, or computer with camera and
microphone.

###  3.2 Patient Protocol

Figure 1 provides a series of screenshots showing the patient at-home or in-
clinic check-in using the ISS application. For each approximately 3-minute
check-in, patients respond to a nine element questionnaire based on the Brief
Pain Inventory (BPI, licensed from MD Anderson) [16, 17, 18] and two prompts
to record videos of themselves.

Figure 1: Submitting a Video through the ISS Mobile Application.

####  3.2.1 Questions 1 - 9: Self-Reported Pain Scores

In addition to the self-reported “Pain Target" which was assigned to each
patient upon enrollment shown in Table 2, there are nine additional self-
reported pain scores. We capture these self-reported pain scores based on the
cancer pain literature which indicates that cancer patients experience complex
emotions and beliefs that can influence their perception of pain and as a
result, clinical treatment [3, 2]. These nine pain scores are submitted at the
time of video submission and change at each submission. They are distinct and
unrelated to the “Pain Target" which is used for the baseline classification
tasks. There is no formula that relates the nine self-reported scores among
themselves or to the “Pain Target". In the below, Question 1 captures current
pain intensity scored on an 11-point Likert Scale (0 No Pain - 10 Worst
Possible Pain) followed by Question 2 which, when answered affirmatively,
indicates the presence of chronic pain. Questions 3-9 utilize an 11-point
Likert Scale (0 Does not Interfere - 10 Completely Interferes) to measure the
interference of pain in an individual’s activity (3, 5, 6) and an individual’s
affect or mood (4, 7, 8, 9).

  1. 1.

How do you rate your pain right now? (0 No Pain – 10 Worst Possible Pain on
Likert-scale).

  2. 2.

Do you have pain, related to your cancer, that has lasted for more than 3
months? (Yes/No)

  3. 3.

How is your pain interfering with your General Activity?

  4. 4.

How is your pain interfering with your Mood?

  5. 5.

How is your pain interfering with your Walking Ability?

  6. 6.

How is your pain interfering with your Normal Work (both work outside the home
and housework)?

  7. 7.

How is your pain interfering with your Relationships with other people?

  8. 8.

How is your pain interfering with your Sleep?

  9. 9.

How is your pain interfering with your Enjoyment of life?

####  3.2.2 Questions 10 and 11: Prompt and Narrative

Following the questionnaire, Question 10 is a prompt to record a video where
the patient reads a 10-15 second passage of text at a grade 3 reading level
selected at random from three different passages. The use of this sort of
prompt is common practice in mood induction or conditioning trials where a
neutral, non-emotion inducing prompt is used as a control versus a
potentially, emotionally charged response related to the experimental
condition [40, 41, 42]. The neutral passage options are:

  * •

“Sarah stepped down from the wagon, a cloth bag in her hand. She reached up
and took off her yellow bonnet, smoothing back her brown hair into a bun. She
was plain and tall.” From Sarah Plain and Tall by Patricia MacLachlan [43]

  * •

“And then the dog came running around the corner. He was a big dog. And ugly.
And he looked like he was having a real good time. His tongue was hanging out
and he was wagging his tail.” From Because of Winn Dixie by Kate DiCamillo
[44]

  * •

“You have brains in your head. You have feet in your shoes. You can steer
yourself any direction you choose. You’re on your own. And you know what you
know. And YOU are the one who’ll decide where to go.” From Oh the Places You
Will Go by Dr. Seuss [45]

Finally, in Question 11, the patient records a video responding to the prompt
_"Please describe how you feel right now.”_ Narratives include discussion of
medical conditions, mood, daily activities, current beliefs and attitudes
about their pain. The allowable video length can range from 15 seconds to 3
minutes, with recording instructions shown prior to each video prompt. For
“at-home” check-ins, patients are instructed to complete the submission alone,
in a quiet and brightly lit room, preferably with a white wall or background.
In addition, patients are asked not to reveal personal information such as
their name or address. In Figure 1, the application screens for Questions 10
and 11 include a live video image to help the patient keep their face centered
in the frame, but the application blurs the video. The blur effect is to
prevent the patient from manipulating their facial expression and minimize
self-conscious alteration of their appearance, allowing them to focus on their
responses.

###  3.3 Data Description

A high level summary of the ISS dataset is provided in Table LABEL:summary.
The ISS dataset is comprised of 29 patients submitting videos in a
spontaneous, non-posed, home setting through a smartphone or computer.
Patients are adults over the age of 18 y.o. and consist of the following
demographics: 20 Male, 9 Female, 17 Skin Type I-III, 12 Skin Type IV - VI. All
patients were enrolled between December 2020 and July 2021. There are 189,999
total video frames. After facial detection, we extracted 173,011 facial
images. After landmark detection on the facial images, the dataset was reduced
by 2.86% to 168,063 facial images with landmarks, since landmarks could not be
detected for some faces. We show the ratio of data imbalance across four pain
levels using the total frames in Table LABEL:summary where the "None" label is
the majority class. The dataset also contains self-reported pain scores from
Questions 1 - 9, described in detail in the Study Design section, along with
sex and skin type labels assigned upon enrollment. Additional descriptive
analysis is provided in Supplementary Materials.

Table 4: ISS Data Summary.

ISS Data Summary Ratios of Total Frames by Pain Levels Total Patients 29 20 M,
9 F, 17 Skin Type I-III, 12 Skin Type IV-VI 4 Pain Levels Frames Ratio No.
Patients 2 Pain Levels Frames Ratio No. Patients Total Videos 509 Avg. Videos
per Patient 17.55 None 100984 1.00 13 No Pain 100984 1.00 13 Total Frames
189,999 Avg. Frames per Patient 6551 Low 11784 8.57 4 Pain 89015 1.13 16 Total
Duration 316 min. Avg. Duration per Patient 655 sec. Mod. 25999 3.88 4 Avg.
Duration per Video 37.32 sec. Range of Duration per Video 3.52 - 135.79 sec.
Severe 51232 1.97 8

Figure 2: Distribution of ISS Data. Histograms for the total videos, frames,
seconds, and average seconds per video, for the ISS dataset are in the four
left-most plots. The four plots on the right illustrate the distribution of
patients (y axis) by the four pain levels, when combined into two pain levels,
by sex, and by skin type.

A notional depiction of ISS data types is shown in Figure 3 to provide context
for the data types. Due to the sensitivity of Personal Identifiable
Information (PII) in the clinical study protocol, we are unable to display
actual facial images from the dataset at this time.

Figure 3: ISS Data Types. Facial images shown are _not_ actual patients from
the ISS dataset due to privacy restrictions. The ISS Dataset currently
consists of six types of data: 1) Nine Self-Reported Pain Scores, 2) Labels
for Sex, Skin Type, and Timeframe, 3) Patient Narrative Video, 4) Video
Extracts: Frames, Faces, Landmarks, and 5) Patient Audio, and 6) Audio
Extracts: Mel Spectrogram, Audio Features.

####  3.3.1 Data Extraction

We use the patient narrative (Question 11) video files (.mp4) and extract
frames at 10 frames-per-second. We decide to use the narrative versus the
prompt since it may contain greater signals of pain and emotion, compared to
the neutral baseline recording. An audio .wav file of the patient narrative is
simultaneously extracted using the ffmpeg library. We use the PyTorch FaceNet
library that implements a fast and CUDA-enabled version of the Multi-task
Cascaded Convolutional Networks (MTCNN) algorithm [46] using an
InceptionResnetV1 model pre-trained on VGGFace2 for face detection and
cropping faces from frames. All patient faces were recorded in a frontally
aligned position so no realignment was implemented. Similar to [26], we
extract features using AAMs. Specifically, we use the Google MediaPipe [47]
Face Mesh AAM model based on 3D Morphable Models [15] to detect facial
landmarks where each face returns an array of 468 points for three
coordinates. From the audio .wav file, we use the Librosa [48] library to
generate a Mel Spectrogram (n_fft=2048, hop_length=512, n_mels=128), and apply
signal processing to capture audio features about the .wav file to include
Mel-frequency cepstral coefficients (MFCCs), chromogram, spectral centroid,
spectral bandwidth, roll-off frequency, and zero crossing rate, leading to 25
audio features. We further break up the original video into 4-second chunks
leading to 40 frames per video chunk, extracting its respective .wav file,
spectrogram, and audio features.

####  3.3.2 No External Labels

In contrast to existing acute pain datasets [7, 8, 9], the ISS dataset lacks
external offline labeling traditionally completed using the Facial Action
Coding System (FACS) [10]. Per the ISS problem statement, the goal is to
predict patient (self)-reported pain, as opposed to observations made by non-
patients via offline pain coders. There are three reasons for not externally
encoding ISS video frames using FACS. First, researchers agree that FACS is
expensive due to the need for a trained coder to annotate each video frame,
making the process time-consuming and clinically infeasible [49, 18]. Second,
ethicists and psychologists argue that there is limited evidence that facial
expressions are reliably and specifically mapped to emotion production [50,
51]. Emotion production is not necessarily tied to a single set of facial
expressions, but relies on the context of the situation and human culture
[50]. Third, cancer patients with chronic pain may not display the typical set
of facial action units (AUs) commonly associated with acute pain. For example
[11] collected video data from 43 outpatient lung cancer patients obtained in
a spontaneous home setting [11]. They found that the cancer patients were more
subdued in expression, and displayed fewer AUs such as grimaces or clenched
teeth, commonly found in facial pain images. As a result, AU labels associated
with pain such as brow lowering (AU4), orbital tightening (AU6, AU7), levator
contraction (AU9, AU10), and eye closure (AU43) may not be applicable to
chronic cancer facial pain detection [52]. However, when the ISS dataset is
released, there are no prohibitions on researchers attempting to annotate
using FACS.

###  3.4 Data Storage and Access

A secure cloud-based environment receives mobile and web-based submissions of
patients’ video, audio, and survey (nine self-reported pain scores) data. No
PII such as names or date of birth is stored, with the exception of face,
voice, and sex information. The environment is AWS GovCloud FedRAMP Moderate,
with Federal Information Security Management Act (FISMA) moderate Authority-
to-Operate (ATO) credentials.

The ISS dataset consists of cancer patients discussing their medical
conditions. The very nature of the images and videos make the data Protected
Health Information (PHI) due to the NIH/NCI not being classified as a "covered
entity". Extreme care must be exercised to ensure patient privacy and rights
are not violated. As a result, we plan to ensure proper patient protections by
placing the collected data in restricted access repositories under the
stewardship of the NIH. Members of the scientific community will be able to
request access to the data and code which may be granted on a per-case basis.
This requirement is necessary to ensure legal requirements are met, avoid
public spillage of PII data, and ensure patient trust that their data is used
within the scope of the intended scientific use. In return researchers receive
access to a dataset with numerous modalities and potential clinical relevance
of results.

##  4 Baselines

We conduct seven baseline experiments for a classification task to predict
each patient’s self-reported “Pain Target" level assigned at the start of
their enrollment shown in Table 2. These levels are fixed upon enrollment for
cohort assignment and remain unchanged throughout the study. As a result, one
patient represents a single pain level throughout the study. All experiments
are static models, which return predictions on a frame-by-frame level. Given
how we are in the initial phase of the ISS study, we train models using facial
images, landmarks, and the additional nine self-reported pain scores for
emotion and activity. However, we do provide baseline results on 4-second
chunks of audio via spectrograms and audio features. These are meant to be
representative of common approaches to similar work, and establish the careful
curation results in a task more difficult than prior literature with simpler
labeling or collection. More details on all results are in the Supplementary
Materials.

Training Details All experiments are trained using 10-fold cross validation
where three test patients are withheld in the test set for nine splits and two
patients set aside for the tenth split. There is no overlap between training
and test sets for each split. Please refer to the Supplementary Materials
Appendix Section F.1. Table 10 that shows the “10-fold-CV details - Test
Patients per Split." For neural networks in Experiments 1 and 3 - 7, we use a
batch size of 16, Adam optimizer with 1e-4 learning rate, and cross entropy as
the loss function, training for 10 epochs, for all experiments. The batch size
of 16 was selected empirically based on cross validation accuracy, after
running several experiments varying batch size from 4, 8, 16, 32, and 64. We
selected Adam optimizer since it has been used in recent facial pain detection
studies such as [24, 53]. We fine-tune ResNet50 as the convolutional neural
network (CNN) backbone for all multimodal experiments, which is pretrained on
ImageNet. We use PyTorch for model training and train on four NVIDIA Tesla T4
GPUs. Experiments 2 and 3 are trained using the Scikit-Learn library for the
Random Forest Classifier, using 100 estimators, gini criterion,
min_samples_split=2, and min_samples_leaf=1.

Experiment 1: Pain Prediction using Static Face Images The first set of
experiments only uses static, facial images. We fine-tune ResNet-50 [54]
pretrained on ImageNet [55] to predict four and two levels of pain. Four
levels are “None" (Self-Reported Pain Level 0), “Low" (1-3), “Moderate" (4-6),
or “Severe" (7-10), and two levels combine “Low", “Moderate", and “Severe"
pain levels into a single “Pain" class. Training binary classifiers for“No
Pain"/“Pain" prediction is similar to many existing facial pain detection
works [7, 22]. We found that the binary classifier leads to better test
patient accuracy scores, and continued Experiments 2 through 7 using only two
pain levels.

Experiments 2, 3: Pain Prediction using Static Landmarks or Pain In these
experiments, we use only one modality to train two separate models and use
traditional machine learning models, specifically the Random Forest algorithm
[56]. Experiment 2 uses the landmark arrays detected for each facial image and
Experiment 3 uses the nine self-reported pain scores explained in Section
3.2.1 that represent how pain interferes with the patient’s emotions and
activity, plus labels for sex, skin type, and timeframe. The timeframe label
is categorical and is extracted from the video submission timestamp
representing what time of day (early AM, late PM, etc.) the video was
submitted. For both Experiments 2 and 3, we train a Random Forest Classifier.
Note, that the target “Pain Target" is not in the set of the nine self-
reported pain scores, which are distinct and separate.

Experiments 4 - 6: Pain Prediction using Static Multimodal Data We train three
multimodal networks using an early, joint fusion strategy as proposed by [57].
For Experiment 4 (“Fusion 1"), we concatenate the fully connected outputs of
ResNet50 with raw landmarks. The feature vector is then inputted to a
feedforward neural network for binary pain prediction. Experiment 5 (“Fusion
2") concatenates the fully connected outputs of ResNet50 with raw landmarks,
in addition to the nine pain scores, skin, sex, and timeframe labels.
Similarly, the feature vector is inputted to the same feedforward network
architecture for binary pain prediction. Experiment 6 (“Fusion 3")
concatenates three vectors: the feature map from layer-4-conv2D-1, the
landmark features outputted from a landmark-specific feedforward network, and
the nine pain scores, sex, skin, and timeframe features outputted from a pain-
specific feedforward network. The resulting feature vector is inputted to a
CNN for binary pain prediction.

Experiment 7: Audio Models Experiment 7 is a binary pain prediction model that
uses the Mel spectrogram image and 25 audio features from 4-second chunks of
audio extracted from each patient video. A feature vector resulting from the
concatenation of the spectrogram feature map from layer-4-conv2D-1 and audio
features learned by a feedforward network, are inputted to the same CNN
architecture as used in Experiment 6. Diagrams for all experimental
architectures are provided in the Supplementary Materials.

##  5 Results

Accuracy Calculation The accuracy of each model is evaluated for each test
patient using the tenth model checkpoint. Using the checkpoint, we evaluate
each test patient individually. We only evaluate test patients using their
respective, assigned split per 10-fold cross-validation (See Supplementary
Materials Section F.1. Table 10 “10-fold-CV details - Test Patients per Split"
for details). For example, test patients 0002, 0029, and 0021 are only
evaluated using the trained model from Split 1, not Split 2 which would have
included these three patients in its training set. We evaluate each test
patient using a batch size of 1, predicting the target pain score for each
patient image. We then calculate accuracy for the test patient in question as
simply $accuracy\\_score(y\\_true,y\\_pred)$ where $y\\_true$ is the set of
true “Pain Target" labels and $y\\_pred$ is the set of predicted “Pain Target"
labels.

As a result, in Table 5, we show the mean accuracy computed for each “Pain
Target" level across all test patients (“No Pain" or “Pain" for two levels,
and “None", “Low", “Moderate", or “Severe" for four levels of pain). For
example, in Experiment 1 “ResNet50-4-static", the accuracy scores for all
patients with ground truth pain labels of “None", were averaged together to
calculate the result of 0.583. In Figures 4 and 5, the bars are color-coded by
the ground truth “Pain Target" level for each patient. The y-axis is the
accuracy predicted for the patient. For example, upon zooming into Figure
4(a), Patient 0029’s (8 marks from the right of the x-axis) ground truth “Pain
Target" level is “No Pain". However, the Experiment 1 static binary model only
predicts it with 0.309 accuracy.

Experiment Results The Experiment 6 multimodal network combining multiple
features from the facial images, landmarks, pain scores, sex, skin, and
timeframe labels performs the best for overall pain classification. Compared
to training on a single modality alone (Experiments 1, 2, 3, 7), Experiment 6
(Fusion 3) shows the best overall class accuracy of 0.657 shown in Table 5.
Fusion 3 also shows the highest accuracy for the “Pain" level at 0.717.
Experiment 6 (Fusion 3) led to 72.4% of test patients exceeding 50% accuracy
per frame as noted in Figure 5(b). However, it ties with Experiment 5
(Fusion2) and Experiment 2 (Random Forest PM) for 51.7% of test patients
achieving over 75% accuracy per Figure 5(a) and Figure 4(d). While the Random
Forest pain model (Figure 5(d)) shows greater “No Pain" accuracy, using only
the self-reported nine pain scores does not detect the original “Low" pain
levels as well as the multimodal Fusion 3 model visualized in Figure 5(c)
shown in blue bars.

Experiment 3 (Random Forest Pain) shows the highest “No Pain" accuracy scores
at 0.706 per Table 5. Adding the nine self-reported pain scores appears to
boost accuracy, compared to training only on faces and landmarks per
Experiment 4 (Fusion 1, 0.513) in Table 5. This is likely due to high
correlations between the nine reported pain scores. Analysis shows strong
Pearson correlation values exceeding 0.89 among activity, mood, work,
enjoyment, and relationship scores. Continued analysis as more patients enroll
in the study is required to understand the effect of the nine pain scores
across all patients. The facial landmarks perform the worst in Experiment 2
(Random Forest LM) with only 37.9% of test patients exceeding better than
random at over 50% accuracy per Figure 4(b). However, when adding landmarks to
facial images in Experiment 4 (Fusion 1), several test patients completely
fail to be detected (1, 2, 16, 13, 29, 3, 28, 25) per Figure 4(d). This may be
consistent with recent research by [34] who show that landmark detection
declines when comparing different populations, such as older patients with
dementia, to healthy adults.

Table 5: Experiment Results by Pain Level Accuracy. “LM" indicates facial
landmarks.

Experiment 4-Class Model Data All Classes None Low Moderate Severe Exp. 1
ResNet50-4-static Faces , only 0.378 0.583 0.168 0.252 0.213 2-Class Model All
Classes No Pain Pain Exp. 1 ResNet50-2-static Faces, only 0.568 0.513 0.612
Exp. 2 Random Forest LM Landmarks, only 0.373 0.479 0.287 Exp. 3 Random Forest
Pain Pain Scores, only 0.650 0.706 0.602 Exp. 4 Fusion 1 Faces + Landmarks
0.513 0.304 0.683 Exp. 5 Fusion 2 Faces + Landmarks + Pain Scores 0.631 0.563
0.687 Exp. 6 Fusion 3 Faces + Landmarks + Pain Scores 0.657 0.582 0.717 Exp. 7
Static Audio Audio, only 0.456 0.645 0.303

(a) Faces, 48.3%

(b) LM, 37.9%

(c) Pain, 65.5%

(d) F1, 51.7%

Figure 4:  Accuracy Scores per Test Patient by Model: Faces, Landmarks, Pain,
and Fusion 1. We show the resulting scores per test patient for the binary
pain classifiers. Horizontal bar indicates 50% accuracy. Percentages in sub-
captions indicates the number of patients exceeding 50% test accuracy.
Notation: Faces=ResNet50-2-static; LM=Random Forest LM (landmarks);
Pain=Random Forest Pain; F1=Fusion1. Best viewed in color and zoomed in.

(a) F2, 62.1%

(b) F3, 72.4%

(c) F3 - 4 Labels

(d) Pain - 4 Labels

Figure 5:  Accuracy Scores per Test Patient by Model - Fusion 2, 3, and Pain,
Visualized with 4 Original Labels. We show the resulting scores per test
patient for the binary pain classifiers. Horizontal bar indicates 50%
accuracy. Percentages in sub-captions indicates the number of patients
exceeding 50% test accuracy. Notation: Pain=Random Forest Pain; F1=Fusion1;
F2=Fusion2; F3=Fusion3. Best viewed in color and zoomed in.

##  6 Discussion and Future Work

Due to a variety of state-of-the-art techniques, we sought to implement simple
models to demonstrate baseline results using fairly minimal preprocessing,
transformations, and architectures. The results of our models show the
dataset’s difficulty. For comparison, acute pain detection studies have shown
accuracy scores up to 82.4% (hit rate) [26] using the UNBC-McMaster Shoulder
Dataset and 95% for multimodal infant pain detection using a custom dataset by
[31]. Chronic pain detection using psychological inventories have achieved
86.5% (cross-validated balanced accuracy) using a support vector machine [58].

Limitations The first limitation of the dataset is the low number of currently
enrolled patients at only 29 patients and the imbalance across pain levels.
However, we observe that two new patients enroll into the study every month.
As the number of patients grow, we expect a more balanced distribution of pain
levels, sex, skin type, and increased volume of data, consistent with the
cohort design indicated in Table 2. However, medical datasets using active
patient populations for major diseases such as cancer, are extremely scarce
due to the time and review required for medical privacy and ethics. This
differs greatly from current pain datasets that have recruited fairly healthy
patients, who are not actively undergoing disease treatment. Due to the
special sensitivity of the ISS study population, we believe that our current
initial results offers important insights currently missing in the medical AI
community.

Next, despite the patient instructions to complete the submission in a quiet,
brightly lit room with a white wall or background, many videos submitted
varied in quality and resolution. The following examples observed in the
dataset present challenges to machine learning: 1) Patient sitting in front of
a door with signage in the background showing letters and numbers; 2) Patient
occasionally wears a mask in some videos (due to Covid-19); 3) Patient records
video in area of intense sunshine and glare causing reflection from various
surfaces; 4) Patient records in a dark, shady room, leading to grainy
resolution and video quality; 5) Patient speaks very quietly or muffled,
making it difficult to hear the patient narrative; 6) Missing data as is the
case of Patient 0009 and absent self-reported nine pain scores from Patient
0015.

Ethics Publicly available acute pain datasets have lacked ethnic diversity.
For example, the UNBC-McMaster Database [6] uses ethnicity as a demographic
indicator where out of the original 129 patients (63 Male, 66 Female), a
minimum of 13.2% (17 patients) consisted of non-Caucasian ethnicity (refer to
Table 1 of [6]). It may be less given how studies using the UNBC-McMaster
dataset have access to data from only 25 out of 129 patients [20, 21, 22, 24].
The BioVid and MIntPAIN datasets provide no information about ethnicity and
race [8, 9]. EmoPAIN contains 22 patients (18 Caucasian, 3 African-American, 1
South-Asian) who are majority white [30]. As a result, we sought to increase
the diversity of enrolled patients by using cohorts that include sex and skin
type specifications. While the Fitzpatrick Skin Type scale was originally
developed for dermatological use, it has recently been criticized for its
conflation with race and ethnicity [59]. It has been found to overestimate the
prevalence of Type IV skin classification in African Americans [60]. The
visual grouping of patients into lighter tones (Skin Types I - III) or darker
tones (Skin Types IV - VI) may be too restrictive and biased in terms of
broadening our diversity of patients. As a result, the ISS dataset requires
careful monitoring and a regular ethics review.

Future Work The second phase of the study will analyze more diverse
modalities. First, we will extract text from the audio files and explore its
utility towards multimodal pain models. Next, since patients were unable to
conduct in-clinic visits, we were unable to gather thermal imagery captured
from a thermal camera stationed at the clinic. Thermal imagery offers insights
into physiological states that is unseen on visible images alone [61]. Our
intent is to generate paired visible-thermal datasets as collected by the
Iris, Eurecom, and Equinox datasets [62, 63, 64]. Lastly, we estimate that
after enrolling 112 patients, the ISS dataset will contain an additional 1,456
videos, 543,733 frames, and 3.8 hours of content.

## References

  * Van den Beuken-van Everdingen et al. [2007] M. Van den Beuken-van Everdingen, J. De Rijke, A. Kessels, H. Schouten, M. Van Kleef, and J. Patijn, “Prevalence of pain in patients with cancer: a systematic review of the past 40 years,” _Annals of oncology_ , vol. 18, no. 9, pp. 1437–1449, 2007.
  * Sun et al. [2007] V. C.-Y. Sun, T. Borneman, B. Ferrell, B. Piper, M. Koczywas, and K. Choi, “Overcoming barriers to cancer pain management: an institutional change model,” _Journal of pain and symptom management_ , vol. 34, no. 4, pp. 359–369, 2007.
  * Cleary [2000] J. F. Cleary, “Cancer pain management,” _Cancer Control_ , vol. 7, no. 2, pp. 120–131, 2000.
  * Lötsch and Ultsch [2018] J. Lötsch and A. Ultsch, “Machine learning in pain research,” _Pain_ , vol. 159, no. 4, p. 623, 2018.
  * Hu et al. [2018] B. Hu, C. Kim, X. Ning, and X. Xu, “Using a deep learning network to recognise low back pain in static standing,” _Ergonomics_ , vol. 61, no. 10, pp. 1374–1381, 2018.
  * Prkachin and Solomon [2008] K. M. Prkachin and P. E. Solomon, “The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain,” _Pain_ , vol. 139, no. 2, pp. 267–274, 2008.
  * Lucey et al. [2011] P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, and I. Matthews, “Painful data: The unbc-mcmaster shoulder pain expression archive database,” in _2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG)_. IEEE, 2011, pp. 57–64.
  * Walter et al. [2013] S. Walter, S. Gruss, H. Ehleiter, J. Tan, H. C. Traue, P. Werner, A. Al-Hamadi, S. Crawcour, A. O. Andrade, and G. M. da Silva, “The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system,” in _2013 IEEE international conference on cybernetics (CYBCO)_. IEEE, 2013, pp. 128–131.
  * Haque et al. [2018] M. A. Haque, R. B. Bautista, F. Noroozi, K. Kulkarni, C. B. Laursen, R. Irani, M. Bellantonio, S. Escalera, G. Anbarjafari, K. Nasrollahi _et al._ , “Deep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities,” in _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 250–257.
  * EkmanW and Friesen [1978] P. EkmanW and V. Friesen, “Facial action coding system manual,” 1978.
  * Wilkie [1995] D. J. Wilkie, “Facial expressions of pain in lung cancer,” _Analgesia_ , vol. 1, no. 2, pp. 91–99, 1995.
  * [12] N. C. I. (NCI), “Machine learning to analyze facial imaging, voice and spoken language for the capture and classification of cancer/tumor pain - full text view,” Jun 2020. [Online]. Available: https://clinicaltrials.gov/ct2/show/NCT04442425
  * Pomeraniec et al. [2020] I. J. Pomeraniec, A. Cha, M. Rule, C. Ordun, M. Hirlinger, and J. Gulley, “Intelligent sight & sound: Machine learning to analyze facial imaging and voice mapping for classification of cancer pain,” _Neurosurgery_ , vol. 67, no. Supplement_1, p. nyaa447_524, 2020.
  * Cootes et al. [2001] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 23, no. 6, pp. 681–685, 2001.
  * Kartynnik et al. [2019] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grundmann, “Real-time facial surface geometry from monocular video on mobile gpus,” _arXiv preprint arXiv:1907.06724_ , 2019.
  * Cleeland [1991] C. S. Cleeland, “Pain assessment in cancer,” _Effect of cancer on quality of life_ , vol. 293, p. 305, 1991.
  * Cleeland and Ryan [1991] C. S. Cleeland and K. Ryan, “The brief pain inventory,” _Pain Research Group_ , pp. 143–147, 1991.
  * Kunz et al. [2017] M. Kunz, D. Seuss, T. Hassan, J. U. Garbas, M. Siebers, U. Schmid, M. Schöberl, and S. Lautenbacher, “Problems of video-based pain detection in patients with dementia: a road map to an interdisciplinary solution,” _BMC geriatrics_ , vol. 17, no. 1, pp. 1–8, 2017.
  * Lucey et al. [2010] P. Lucey, J. F. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, and K. M. Prkachin, “Automatically detecting pain in video through facial action units,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 41, no. 3, pp. 664–674, 2010.
  * Zhou et al. [2016] J. Zhou, X. Hong, F. Su, and G. Zhao, “Recurrent convolutional neural network regression for continuous pain intensity estimation in video,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2016, pp. 84–92.
  * Neshov and Manolova [2015] N. Neshov and A. Manolova, “Pain detection from facial characteristics using supervised descent method,” in _2015 IEEE 8th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS)_ , vol. 1. IEEE, 2015, pp. 251–256.
  * Khan et al. [2013] R. A. Khan, A. Meyer, H. Konik, and S. Bouakaz, “Pain detection through shape and appearance features,” in _2013 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2013, pp. 1–6.
  * Lo Presti and La Cascia [2015] L. Lo Presti and M. La Cascia, “Using hankel matrices for dynamics-based facial emotion recognition and pain detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2015, pp. 26–33.
  * Bellantonio et al. [2016] M. Bellantonio, M. A. Haque, P. Rodriguez, K. Nasrollahi, T. Telve, S. Escalera, J. Gonzalez, T. B. Moeslund, P. Rasti, and G. Anbarjafari, “Spatio-temporal pain recognition in cnn-based super-resolved facial images,” in _Video Analytics. Face and Facial Expression Recognition and Audience Measurement_. Springer, 2016, pp. 151–162.
  * Hassan et al. [2019] T. Hassan, D. Seuß, J. Wollenberg, K. Weitz, M. Kunz, S. Lautenbacher, J.-U. Garbas, and U. Schmid, “Automatic detection of pain from facial expressions: a survey,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 6, pp. 1815–1831, 2019.
  * Ashraf et al. [2009] A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon, “The painful face–pain expression recognition using active appearance models,” _Image and vision computing_ , vol. 27, no. 12, pp. 1788–1796, 2009.
  * Bartlett et al. [2006] M. S. Bartlett, G. Littlewort, M. G. Frank, C. Lainscsek, I. R. Fasel, J. R. Movellan _et al._ , “Automatic recognition of facial actions in spontaneous expressions.” _J. Multim._ , vol. 1, no. 6, pp. 22–35, 2006\.
  * Gruss et al. [2019] S. Gruss, M. Geiger, P. Werner, O. Wilhelm, H. C. Traue, A. Al-Hamadi, and S. Walter, “Multi-modal signals for analyzing pain responses to thermal and electrical stimuli,” _JoVE (Journal of Visualized Experiments)_ , no. 146, p. e59057, 2019.
  * Werner et al. [2014] P. Werner, A. Al-Hamadi, R. Niese, S. Walter, S. Gruss, and H. C. Traue, “Automatic pain recognition from video and biomedical signals,” in _2014 22nd International Conference on Pattern Recognition_. IEEE, 2014, pp. 4582–4587.
  * Aung et al. [2015] M. S. Aung, S. Kaltwang, B. Romera-Paredes, B. Martinez, A. Singh, M. Cella, M. Valstar, H. Meng, A. Kemp, M. Shafizadeh _et al._ , “The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal emopain dataset,” _IEEE transactions on affective computing_ , vol. 7, no. 4, pp. 435–451, 2015.
  * Zamzmi et al. [2016] G. Zamzmi, C.-Y. Pai, D. Goldgof, R. Kasturi, T. Ashmeade, and Y. Sun, “An approach for automated multimodal analysis of infants’ pain,” in _2016 23rd International Conference on Pattern Recognition (ICPR)_. IEEE, 2016, pp. 4148–4153.
  * Zamzmi et al. [2018] G. Zamzmi, D. Goldgof, R. Kasturi, and Y. Sun, “Neonatal pain expression recognition using transfer learning,” _arXiv preprint arXiv:1807.01631_ , 2018.
  * Rezaei et al. [2020] S. Rezaei, A. Moturu, S. Zhao, K. M. Prkachin, T. Hadjistavropoulos, and B. Taati, “Unobtrusive pain monitoring in older adults with dementia using pairwise and contrastive training,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 25, no. 5, pp. 1450–1462, 2020.
  * Asgarian et al. [2019] A. Asgarian, S. Zhao, A. B. Ashraf, M. E. Browne, K. M. Prkachin, A. Mihailidis, T. Hadjistavropoulos, and B. Taati, “Limitations and biases in facial landmark detection d an empirical study on older adults with dementia.” in _CVPR Workshops_ , 2019, pp. 28–36.
  * Salekin et al. [2021] M. S. Salekin, G. Zamzmi, D. Goldgof, R. Kasturi, T. Ho, and Y. Sun, “Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment,” _Computers in Biology and Medicine_ , vol. 129, p. 104150, 2021.
  * Salekin et al. [2019] M. S. Salekin, G. Zamzmi _et al._ , “Multi-channel neural network for assessing neonatal pain from videos,” in _2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)_. IEEE, 2019, pp. 1551–1556.
  * Othman et al. [2021] E. Othman, P. Werner, F. Saxen, A. Al-Hamadi, S. Gruss, and S. Walter, “Automatic vs. human recognition of pain intensity from facial expression on the x-ite pain database,” _Sensors_ , vol. 21, no. 9, p. 3273, 2021.
  * Goldsmith et al. [2012] L. Goldsmith, S. Katz, B. Bilchrest, A. Paller, D. Leffel, and K. Wolff, “Fitzpatrick’s dermatology in general medicine, ed,” _McGrawHill Medical_ , pp. 2421–2429, 2012.
  * Haefeli and Elfering [2006] M. Haefeli and A. Elfering, “Pain assessment,” _European Spine Journal_ , vol. 15, no. 1, pp. S17–S24, 2006.
  * Apolinário-Hagen et al. [2018] J. Apolinário-Hagen, L. Fritsche, C. Bierhals, and C. Salewski, “Improving attitudes toward e-mental health services in the general population via psychoeducational information material: A randomized controlled trial,” _Internet interventions_ , vol. 12, pp. 141–149, 2018.
  * Fink-Lamotte et al. [2020] J. Fink-Lamotte, A. Widmann, J. Fader, and C. Exner, “Interpretation bias and contamination-based obsessive-compulsive symptoms influence emotional intensity related to disgust and fear,” _PloS one_ , vol. 15, no. 4, p. e0232362, 2020.
  * Livesay and Porter [1994] J. R. Livesay and T. Porter, “Emg and cardiovascular responses to emotionally provocative photographs and text,” _Perceptual and motor skills_ , vol. 79, no. 1, pp. 579–594, 1994.
  * MacLachlan and Beech [1997] P. MacLachlan and L. Beech, _Sarah, plain and tall_. Scholastic Inc., 1997.
  * DiCamillo [2009] K. DiCamillo, _Because of Winn-Dixie_. Candlewick Press, 2009.
  * Geisel [1990] T. S. Geisel, _Oh, the places you’ll go!_ Random House Books for Young Readers, 1990.
  * Zhang et al. [2016] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * Lugaresi et al. [2019] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee _et al._ , “Mediapipe: A framework for building perception pipelines,” _arXiv preprint arXiv:1906.08172_ , 2019\.
  * McFee et al. [2021] B. McFee, A. Metsai, M. McVicar _et al._ , “librosa/librosa: 0.8.1rc2,” May 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4792298
  * Chen et al. [2018] Z. Chen, R. Ansari, and D. Wilkie, “Automated pain detection from facial expressions using facs: A review,” _arXiv preprint arXiv:1811.07988_ , 2018\.
  * Barrett et al. [2019] L. F. Barrett, R. Adolphs, S. Marsella, A. M. Martinez, and S. D. Pollak, “Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements,” _Psychological science in the public interest_ , vol. 20, no. 1, pp. 1–68, 2019.
  * Crawford [2021] K. Crawford, _The Atlas of AI_. Yale University Press, 2021.
  * Prkachin [1992] K. M. Prkachin, “The consistency of facial expressions of pain: a comparison across modalities,” _Pain_ , vol. 51, no. 3, pp. 297–306, 1992.
  * Rodriguez et al. [2017] P. Rodriguez, G. Cucurull, J. Gonzàlez, J. M. Gonfaus, K. Nasrollahi, T. B. Moeslund, and F. X. Roca, “Deep pain: Exploiting long short-term memory networks for facial expression classification,” _IEEE transactions on cybernetics_ , 2017.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * Breiman [2001] L. Breiman, “Random forests,” _Machine learning_ , vol. 45, no. 1, pp. 5–32, 2001.
  * Huang et al. [2020] S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren, “Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines,” _NPJ digital medicine_ , vol. 3, no. 1, pp. 1–9, 2020.
  * Antonucci et al. [2020] L. A. Antonucci, A. Taurino, D. Laera, P. Taurisano, J. Losole, S. Lutricuso, C. Abbatantuono, M. Giglio, M. F. De Caro, G. Varrassi _et al._ , “An ensemble of psychological and physical health indices discriminates between individuals with chronic pain and healthy controls with high reliability: A machine learning study,” _Pain and Therapy_ , vol. 9, no. 2, pp. 601–614, 2020.
  * Ware et al. [2020] O. R. Ware, J. E. Dawson, M. M. Shinohara, and S. C. Taylor, “Racial limitations of fitzpatrick skin type,” _Cutis_ , vol. 105, no. 2, pp. 77–80, 2020.
  * Pichon et al. [2010] L. C. Pichon, H. Landrine, I. Corral, Y. Hao, J. A. Mayer, and K. D. Hoerster, “Measuring skin cancer risk in african americans: is the fitzpatrick skin type classification scale culturally sensitive,” _Ethn Dis_ , vol. 20, no. 2, pp. 174–179, 2010.
  * Ordun et al. [2020] C. Ordun, E. Raff, and S. Purushotham, “The use of ai for thermal emotion recognition: A review of problems and limitations in standard design and data,” _arXiv preprint arXiv:2009.10589_ , 2020.
  * Selinger et al. [2006] A. Selinger _et al._ , “Appearance-based facial recognition using visible and thermal imagery: a comparative study,” Equinox Corp., Tech. Rep., 2006.
  * [63] Otcbvs benchmark dataset collection. [Online]. Available: http://vcipl-okstate.org/pbvs/bench/
  * Mallat et al. [2018] K. Mallat _et al._ , “A benchmark database of visible and thermal paired face images across multiple variations,” in _BIOSIG_. IEEE, 2018, pp. 1–5.
  * Buolamwini and Gebru [2018] J. Buolamwini and T. Gebru, “Gender shades: Intersectional accuracy disparities in commercial gender classification,” in _Conference on fairness, accountability and transparency_. PMLR, 2018, pp. 77–91.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [No]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [N/A]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [N/A]

    2. (b)

Did you mention the license of the assets? [Yes]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to patients and
screenshots, if applicable? [Yes]

    2. (b)

Did you describe any potential patient risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [Yes]

    3. (c)

Did you include the estimated hourly wage paid to patients and the total
amount spent on patient compensation? [Yes]

##  Appendix A Author Statement

Upon future release of the data, NIH bears all responsibility in case of
violation of rights, etc., and confirmation of the data license. Wherever
possible, all data are de-identified; however, data consisting of facial image
frames are inherently classified as Personally Identifying Information (PII)
and as such are unable to be de-identified. All data is stored under the
control of an internal NIH database accessible to other NIH researchers for
future research. NIH will share access to outside researchers under their
discretion. As part of the informed consent process, governed by the NIH
Institutional Review Board (IRB), patients have given permission for their
data to be shared under these restricted conditions for future research.

##  Appendix B Acknowledgements

We are grateful for the support and advisement received on the development of
the ISS Study Design and Protocol. We acknowledge Mason Rule and Susan
Wroblewski at the National Institutes of Health, and Lauren Neal, Jeremy
Walsh, and John Larson at Booz Allen Hamilton.

##  Appendix C Ethical Considerations

The generation of this dataset has brought multiple ethical concerns to our
attention. This section is intended to share our observation of ethical
concerns to raise awareness of the issues and stimulate conversation around
potential mitigation. We will discuss the following ethical concerns:

  * •

Use of Fitzpatrick Skin Scale

  * •

Data collection biases

  * •

Video lighting

  * •

Camera quality and availability

  * •

Facial Recognition Algorithm Bias and Surveillance

Use of Fitzpatrick Skin Scale The Fitzpatrick Skin Scale was originally
developed for dermatological use to measure the sensitivity of skin burn
during phototherapy and has recently been critiqued for its conflation with
race and ethnicity [59]. Further, the scale has been found to overestimate the
prevalence of Type IV skin classification in African Americans [60]. In a
sample of 2,086 California Black adults, 59% categorized themselves as not
applicable to any of the four skin types and only 26.8% identified as Type IV,
the darkest skin type. The authors assert that the Fitzpatrick Skin Scale is
more in line with Caucasian experiences of sun-reactivity and not Asians,
Arabs, and African Americans. The visual grouping of patients into “Dark"
(Skin Types IV - VI) or “Light" (Skin Types I - III) may be too restrictive
and biased in terms of broadening our diversity of patients.

Data Collection Biases As we collected videos from patient owned devices (e.g.
mobile phones, tablets, etc), we immediately noticed wide ranges of video
characteristics. Videos were captured using various illuminations (e.g., front
lighting vs. backlighting), varying instances of whether a patient is (or is
not) wearing glasses, and masks are occasionally worn in videos due to the
global Covid-19 pandemic. We also noticed a variety of video quality and
resolution which is most attributed to the condition of the physical device
hardware used to capture the video. As we continue to use this data to train
models, it may be useful to quantify the extent of these many video
characteristics and store them as metadata. Having the video characteristics
stored as metadata would allow for bias detection techniques to be for
identification and (if possible) mitigation of data biases. Although an
analysis of socioeconomic backgrounds of each patient has not been conducted,
it is fathomable that those in lower socioeconomic groups may not have access
to the latest smartphones and devices. This may lead to lower video quality
and as a result, model results that are not representative across the broad
spectrum of patients. Intrinsic penalization based on video quality conflated
with socioeconomic status should be avoided.

Facial Recognition Algorithm Bias and Surveillance Existing bias is always a
concern when using pretrained models for either direct inference or transfer
learning. Facial recognition algorithms are no different. Researchers [65]
explore issues with facial recognition algorithms explored through the context
of the (already concerning) Fitzpatrick Skin Scale. Further, the research
towards chronic cancer facial pain detection is not intended for surveillance
purposes. A potential misuse would be for monetary purposes of estimating
medical risk and liability by analyzing and estimating pain without consent.

##  Appendix D Preprocessing

###  D.1 Corrupt Data

Patient 0009’s videos were unreadable and could not be processed. This
consists of six potential video submissions. Patient 0015 submitted four
videos, but never submitted pain scores at the start of the submission. As a
result, there are no pain feature responses for Patient 0015. We set all nine
self-reported pain scores for Patient 0015 to zero.

###  D.2 Key Processing Steps

We describe preprocessing and encoding for each of the six datatypes in the
ISS Dataset: 1) Self-Reported Pain Scores, 2) Labels for Sex, Skin Tone, and
Timeframe, 3) Patient Narrative Video, 4) Video Extracts: Frames, Faces,
Landmarks, and 5) Patient Audio, and 6) Audio Extracts: Mel Spectrogram, Audio
Features.

#### D.2.1

Pain Scores, and Sex, Skin Tone, Timeframe Labels At the start of each video
submission, the patient answers a survey that allows them to input self-
reported pain scores for nine questions. No preprocessing is done for the pain
scores which are reported on an 11-point Likert-scale or for the “painnow"
variable, Boolean.

  * •

Patient surveys are converted to .csv files.

  * •

Labels are encoded for Sex: Male: 0, Female: 1 and Skin Tone: I - III: 1, IV -
VI: 0.

  * •

The “timeframe" is manually extracted from the video submission timestamp
which is reported like 20201201T003232. We encode the hour code (the two
characters after “T") as the following: early_am = [’05’, ’06’, ’07’], reg_am
= [’08’, ’09’, ’10’, ’11’], early_pm = [’12’, ’13’, ’14’, ’15’, ’16’], reg_pm
= [’17’, ’18’, ’19’, ’20’], early_nite = [’21’, ’22’, ’23’, ’24’], late_nite =
[’00’, ’01’, ’02’, ’03’, ’04’].

####  D.2.2 Videos and Extracts

  * •

Process video for all basic datatypes \- This will extract all frames from all
videos. Do this for one patient at a time per usage instructions below.
Running this script will create directories for each patient per video, with
corresponding frames, .wav files, Mel Spectrogram for each .wav file, and the
encoded csv from the survey. Frames are stored like:
frames/0001/20201201T003232/1.jpg. Audio is stored as:
audio/0001/20201201T003232/20201201T003232.wav. Mel spectograms are stored as
audio/0001/20201201T003232.png. Surveys for pain scores are stored as
surveys/0001/20201201T003232/targets.csv.

  * •

Detect faces on spliced frames \- We use Facenet-Pytorch to detect faces and
crop them directly from the frame to a 160 x 160 image. They are stored in a
sub-directory per patient. This saves crops in the form with a prefix
associated to each frame number crops/0001/0001_20201201T003232_1.jpg.

  * •

Detect and save facial landmarks \- We use the Google Media Pipe API to
extract 468 landmarks for each detected face. Landmarks are stored as a list
of arrays for each face, in a pickle file. File names for all files traversed,
all files where faces were detected, and all files names where faces were not
detected, are also stored as pickle files. The following 4 pickled files are
stored for each patient: landmarks/0001/all_files.pkl,
landmarks/0001/all_landmarks.pkl, landmarks/0001/face_files.pkl,
landmarks/0001/no_face_files.pkl. The “all_landmarks.pkl" holds all the
landmarks as a list of arrays. The other files keep track of what files were
iterated over. “face_files.pkl" is important because this will allow us to
keep track of the exact mapping of 1:1 array:image.

####  D.2.3 Audio and Extracts

Break the large .wav file into smaller 4 second chunks. For each smaller .wav
file, generate its own spectrogram and CSV file of acoustic features.

  * •

Extract statistics about the .wav audio file \- After the .wav file has been
extracted from the patient video, we extract audio features. This will
generate a .csv file labeled per sub-directory for each .wav file. The will
store a single .csv file with all audio files labeled by video timestamp. As a
result, for each patient, there will be a single “audio_features.csv" file
that stores 25 audio features for each patient video. For example, if Patient
A submits 12 times, there will be a .csv file with 12 rows, one for each audio
file extracted per video.

  * •

Audio Features \- 25 features are extracted using the Librosa library. These
include chromogram, spectral centroid (center of mass), spectral rolloff
(signal shape), spectral bandwidth, zero-crossing rate (smoothness), and Mel
Frequency Cepstral Coefficients (MFCCs) (n=20 MFCC) statistics.

  * •

Audio Chunks \- We selected an arbitrary number of seconds at 4-seconds to
evaluate our experiments. However, any number of seconds can be selected to
decompose the original audio file extracted from the basic processing into
smaller chunks. We use the Python PyDub library and select chunk_length_ms =
4000 for 4 second chunks, that do not overlap.

####  D.2.4 Labels

Three sets of labels are generated for each of the ten splits. After the
training data and test data are assigned to each of the ten splits by patient,
we encode the pain class target as either binary (“No Pain", “Pain") or keep
the original four labels. Labels are read in via PyTorch dataloader, specific
to each of the seven experiments. Labels are:

Labels of Images with Landmarks \- Image filename, pain label, sex label, skin
tone label, landmarks.

Figure 6: Example of Images with Landmarks Labels

Labels of Images with Landmarks and Pain Scores \- All data from “Labels of
Images with Landmarks" in addition to pain scores, sex, skin tone, and
timeframe labels.

Figure 7: Example of Images, Landmarks, Pain Scores Labels

Labels of Audio Spectogram Images with Audio Features \- Contains the filename
for the spectrogram and its corresponding set of 25 audio features. Each
spectrogram is generated per audio file. Both the original audio file and the
4-second audio chunks are labeled in this manner.

Figure 8: Example of Audio Labels

####  D.2.5 Model Transforms

We use PyTorch V1.6.0 for implementing transforms. Limited transforms were
conducted to only include RandomResizedCrop() and RandomHorizontalFlip. All
images for the facial images and the Mel spectrogram were resized to 224 x
224. Face crops were originally 160 x 160 following MTCNN detection on frames.
Images were normalized to ([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]). No
image augmentation or synthetic images were used in training.

##  Appendix E Additional Data Analysis

In Table 6 we provide counts for the total number of videos, video frames,
seconds of video, average seconds per video, and 4-sec. chunk .wav files for
each of the 29 patients. Counts are also provided by the original four levels
of pain (“None", “Low", “Moderate", “Severe"), in addition to the two levels
(“No Pain", “Pain") used for the binary classification models.

Table 6: Descriptive Statistics for ISS Data Across 29 Enrolled Patients. The
number of Pain Score submissions per patient is equivalent to the number of
Videos, since pain scores are submitted at the start of each submission.

Patient Videos Total Frames Total Seconds Avg Sec/Video 4 Sec. Chunks (4-Pain)
Level (2-Pain) Level Sex Skin Tone 1 23 1674 167.23 7.27 53 None No Pain M IV
- VI 2 23 1721 172.24 7.49 54 None No Pain M I - III 3 5 2558 255.91 51.18 67
None No Pain M I - III 4 5 1605 160.08 32.02 43 Moderate Pain M I - III 5 5
2552 255.17 51.03 67 Moderate Pain M I - III 6 1 193 19.39 19.39 5 Severe Pain
F IV - VI 7 26 5398 539.61 20.75 148 Low Pain M IV - VI 9 n/a n/a n/a n/a n/a
Low Pain F I - III 8 25 6938 694.21 27.77 185 Moderate Pain F I - III 10 18
3523 352.43 19.58 95 Severe Pain F IV - VI 11 26 35249 3525.39 135.59 896
Severe Pain M I - III 12 55 41373 4136.35 75.21 1064 None No Pain M I - III 13
28 6953 694.98 24.82 187 None No Pain M I - III 14 43 16685 1669.08 38.82 435
None No Pain M I - III 15 4 1435 143.57 35.89 38 Severe Pain F IV - VI 16 16
2438 243.61 15.23 66 None No Pain M I - III 17 33 12556 1253.93 38 328 None No
Pain F I - III 18 3 2779 278.08 92.69 71 None No Pain M I - III 19 18 635
63.38 3.52 21 Severe Pain F I - III 20 8 3222 322.18 40.27 84 Low Pain M I -
III 21 50 14904 1489 29.78 397 Moderate Pain F I - III 22 11 5413 541.28 49.21
141 Severe Pain F I - III 23 9 3711 371.34 41.26 98 Severe Pain M IV - VI 24
17 5332 533.44 31.38 142 None No Pain F I - III 25 15 3504 350.6 23.37 95 None
No Pain M IV - VI 26 4 1073 107.43 26.86 28 Severe Pain M IV - VI 27 9 1478
147.93 16.44 40 None No Pain M IV - VI 28 19 1467 146.48 7.71 44 Low Pain M IV
- VI 29 6 1933 193.27 32.21 51 None No Pain M IV - VI 30 4 1697 169.73 42.43
45 Low Pain M IV - VI

Figure 9: Plots of Descriptive Statistics. Plots are consistent with Table 6 shown above. Figure 10: Correlation Analysis of Self-Reported Pain Scores Table 7: Source Devices Used to Submit Data by each Patient Patient | Source Device
---|---
0001 | Android
0002 | iOS
0003 | iOS
0004 | iOS
0005 | Clinic Only
0006 | Clinic Only
0007 | Android
0008 | iOS
0009 | Android
0010 | iOS
0011 | iOS
0012 | Android
0013 | Android
0014 | iOS
0015 | iOS
0016 | Android
0017 | Android
0018 | iOS
0019 | Android
0020 | iOS
0021 | Android
0022 | iOS
0023 | iOS
0024 | iOS
0025 | iOS
0026 | Android
0027 | iOS
0028 | iOS
0029 | iOS

##  Appendix F Details of the Models

###  F.1 Cross Validation Details

We use 10-fold-cross-validation leading to 10 different training splits, where
9 of the splits reserve 3 patients for the test set and the tenth split uses 2
patients, since there are 29 patients.

Table 8: 10-fold-CV details - Facial Image Counts per Split.

Face Images Train Set Test Set Total Data Split Face Images Face Images w/LMs
Face Images Face Images w/LMs Face Images Face Images w/LMs 1 157269 153322
15742 14741 173011 168063 2 163128 158310 9883 9753 173011 168063 3 163734
158809 9277 9254 173011 168063 4 162678 157865 10333 10198 173011 168063 5
156622 151809 16389 16254 173011 168063 6 168339 163578 4672 4485 173011
168063 7 145522 141804 27489 26259 173011 168063 8 103418 100462 69593 67601
173011 168063 9 166739 161816 6272 6247 173011 168063 10 169650 164792 3361
3271 173011 168063

Table 9: 10-fold-CV details - 4-Second Chunks of Mel Spectrograms and Audio
Features, Counts per Split.

Split Train Set Test Set Total 1 4486 502 4988 2 4719 269 4988 3 4742 246 4988
4 4710 278 4988 5 4561 427 4988 6 4858 130 4988 7 4264 724 4988 8 2841 2147
4988 9 4821 167 4988 10 4890 98 4988

Table 10: 10-fold-CV details - Pain Scores that include Skin, sex, and
Timeframe Labels, Counts per Split.

Split Train Set Test Set Total 1 455 82 537 2 479 58 537 3 505 32 537 4 511 26
537 5 497 40 537 6 495 42 537 7 449 88 537 8 425 112 537 9 510 27 537 10 507
30 537

Table 11: 10-fold-CV details - Test Patients per Split. We show each patient
held out in the test set for each split. For each test patient, the table
shows the original pain level and the binary pain level in order of the
patient. For example, for Split 1 Test Patient 0002, the original pain level
is “Low" and the binary pain level is “Pain".

Split Test Patients Respective Original Pain Level Respective Binary Pain
Level 1 0002 0029 0021 Low None Moderate Pain No Pain Pain 2 0028 0027 0008
Low None Moderate Pain No Pain Pain 3 0020 0025 0005 Low None Moderate Pain No
Pain Pain 4 0015 0024 0023 Severe None Severe Pain No Pain Pain 5 0018 0017
0026 None None Severe No Pain No Pain Pain 6 0004 0016 0019 Moderate None
Severe Pain No Pain Pain 7 0014 0022 0007 None Severe Low No Pain Pain No Pain
8 0013 0011 0012 None Severe None No Pain Pain No Pain 9 0003 0010 0006 None
Severe Severe No Pain Pain Pain 10 0030 0001 None None No Pain No Pain

###  F.2 Model Architectures

Diagrams provide model architecture details for Experiments 4 - 7. The
architecture for Experiment 1 is ResNet50, unmodified, pre-trained on ImageNet
and fine-tuned. Models used for Experiments 2 and 3 are Random Forest
Classifiers using all default settings in the
sklearn.ensemble.RandomForestClassifier library with 100 estimators

Figure 11: Experiment 4: Fusion 1, Faces + Landmarks. Figure 12: Experiment 5:
Fusion 2, Faces + Landmarks + Pain Scores. Figure 13: Experiment 6: Fusion 3,
Faces + Landmarks + Pain Scores. Figure 14: Experiment 7: Static Audio, Audio
only.

##  Appendix G Additional Model Results

We provide additional results from baseline experiments. In Table 12, the
number and percentage of patients with accuracy scores in the ranges of 0 -
25%, 25% - 50%, 50% - 75%, and 75% is shown. In Figure 15, we show the test
patient accuracy scores by binary labels as well as the original four pain
labels. There is an imbalance of pain levels in the ISS dataset, where “None"
frames outnumber “Low" and “Moderate" classes at a ratio of 8.50:1 and 3.88:1,
respectively. When inspecting the binary classifier according to the original
four pain labels, the “Moderate" test patients show greater accuracy. In
Figure 16 we see a decrease in the accuracy of the “Moderate" and “Low" test
patients. In Figure 17, while the “Moderate" classes exceed 50% accuracy, the
model struggles with “Low" pain test patients. However, 13 test patients reach
test accuracies close to 1.0.

Table 12: Model Results by Patient Scores by Accuracy Range.

Test Accuracy Ranges [0. 0.25) [0.25, 0.50) [0.50, 0.75) [0.75, 1.0) Over 50%
Experiments Model No. Patients Perc. Patients No. Patients Perc. Patients No.
Patients Perc. Patients No. Patients Perc. Patients No. Patients Perc.
Patients Exp. 1 ResNet50-4-static 13 44.8% 7 24.1% 3 10.3% 6 20.7% 9
0.31034483 Exp. 1 ResNet50-2-static 6 20.7% 9 31.0% 1 3.4% 13 44.8% 14
0.48275862 Exp. 2 Random Forest LM 15 51.7% 3 10.3% 5 17.2% 6 20.7% 11
0.37931034 Exp. 3 Random Forest Pain 7 24.1% 2 6.9% 4 13.8% 15 51.7% 19
0.65517241 Exp. 4 Fusion 1 11 37.9% 3 10.3% 4 13.8% 11 37.9% 15 0.51724138
Exp. 5 Fusion 2 8 27.6% 3 10.3% 3 10.3% 15 51.7% 18 0.62068966 Exp. 6 Fusion 3
7 24.1% 1 3.4% 6 20.7% 15 51.7% 21 0.72413793 Exp. 7 Static Audio 9 31.0% 7
24.1% 7 24.1% 6 20.7% 13 0.44827586

(a) ResNet50-4-static

(b) ResNet50-2-static - Binary

(c) ResNet50-2-static - 4 Labels

Figure 15:  Accuracy per Patient for Static Face Models. Patients are ranked
in descending order of test accuracy and color coded by pain labels. (a) shows
the results of Experiment 1 (ResNet50-4-static). (b) Experiment 1
(ResNet50-2-static) binary pain classification results. (c) Experiment 1
(ResNet50-4-static) results but visualized with the original four pain labels.

(a) Random Forest LM - Binary

(b) Random Forest LM - 4 Labels

Figure 16:  Accuracy per Patient for Landmark Models.Patients are ranked in
descending order of test accuracy and color coded by pain labels. (a) shows
the results of Experiment 2 (Random Forest LM) binary pain classification
results. (b) shows Experiment 2 (Random Forest LM) results but visualized with
the original four pain labels.

(a) Random Forest Pain - Binary

(b) Random Forest Pain - 4 Labels

Figure 17:  Accuracy per Patient for Pain Models. Patients are ranked in
descending order of test accuracy and color coded by pain labels. (a) shows
the results of Experiment 3 (Random Forest Pain) binary pain classification
results. (b) shows Experiment 3 (Random Forest Pain) results but visualized
with the original four pain labels.

(a) Fusion 1 - Binary

(b) Fusion 2 - Binary

(c) Fusion 3 - Binary

(d) Fusion 1 - 4 Labels

(e) Fusion 2 - 4 Labels

(f) Fusion 3 - 4 Labels

Figure 18:  Accuracy per Patient for Multimodal Face, Landmark and Pain Score
Models. Each subplot shows the results of Experiments 4, 5, and 6, with the
binary predictions as well as visualized with the original four pain levels to
provide greater granularity. Plots (a) and (d) provide accuracy scores for
each test patient for Experiment 4 that combines facial images and landmarks.
Plots (b) and (e) provide accuracy scores for Experiment 5 that combines
facial images, raw landmarks, and raw pain scores, with sex, skin tone, and
timeframe labels by simple concatenations. Plots (c) and (f) provide results
of Experiment 6 that combines learned features from separate networks for each
modality of the facial images, landmarks, and pain scores with sex, skin tone,
and timeframe labels, into a single multimodal network.

(a) Audio, 2 Labels

(b) Audio, 4 Labels

Figure 19:  Accuracy per Patient for Audio Models. Table 13: Bottom Five Test Patients for Each Model, with Near Zero Accuracy. Experiments | Model | Bottom 5 Test Patients
---|---|---
Exp. 1 | Resnet50-2-static | 7, 25, 28, 26, 13
Exp. 2 | Random Forest LM | 8, 20, 19, 28, 17
Exp. 3 | Random Forest Pain | 15, 30, 10, 6, 28
Exp. 4 | Fusion 1 | 1, 2, 16, 13, 29, 25*
Exp. 5 | Fusion 2 | 4, 20, 29, 17, 1
Exp. 6 | Fusion 3 | 4, 27, 29, 25, 20

##  Appendix H Datasheets for Datasets

###  H.1 Motivation

For what purpose was the dataset created? The ISS dataset was created to
explore whether facial video, images, audio, text, and pain scores, could be
used to predict chronic cancer self-reported pain across a diverse patient
group. [Paper section reference: Introduction]

Who created the dataset? The National Institutes of Health (NIH) National
Cancer Institute (NCI). [Paper section reference: Introduction]

Who funded the creation of the dataset? The National Institutes of Health
(NIH) National Cancer Institute (NCI). [Paper section reference: Introduction]

Any other comments? [N/A]

###  H.2 Composition

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description. The ISS dataset consists of 29 cancer
patients currently undergoing active treatment at the NIH, scoring their self-
reported pain levels and narrating their feelings about pain. [Paper section
reference: ISS Dataset/Sample and Study Design]

How many instances are there in total (of each type, if appropriate)? 29
patients are currently enrolled in the study. [Paper section reference: ISS
Dataset/Sample and Study Design]

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? It is the current number
of enrolled patients to date. The larger set will encompass 112 patients once
the study has concluded. [Paper section reference: ISS Dataset/Sample and
Study Design]

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? Each patient submits a video through their smartphone or
home computer. The video is extracted into video frames, facial images, facial
landmarks, audio files, spectrograms, and audio features. [Paper section
reference: ISS Dataset/Data Description]

Is there a label or target associated with each instance? Yes, the label is
“worst pain" which represents the patient’s pain level reported in the past 30
days upon enrollment in the study. This label does not change through the
patient’s time in the study. [Paper section reference: ISS Dataset/Sample and
Study Design; Baselines]

Is any information missing from individual instances? Yes, Patient 0015 lacks
self-reported pain scores, and Patient 0009 video data was unusable. [Paper
section reference: ISS Dataset/Sample and Study Design]

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? N/A.

Are there recommended data splits (e.g., training, development/validation,
testing)? Yes. We use 10-fold-cross-validation that withholds three patients
from nine of the splits, and two patients from the tenth split. [Paper section
reference: Appendix/Details of the Models/Cross Validation Details]

Are there any errors, sources of noise, or redundancies in the dataset?
Patients are inconsistent in their video recording, leading to noisy
backgrounds such as low illumination, muffled or quiet speaking, letters and
signage in the background, shiny bright sunlight and glare, and occasional
mask wearing due to Covid-19. [Paper section reference: Discussion and Future
Work]

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g.,websites, tweets, other datasets)? The ISS dataset
will be self-contained and hosted under the supervision of the NIH. [Paper
section reference: ISS Dataset/Data Storage and Access]

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor–patient
confidentiality, data that includes the content of individuals’ non-public
communications)? Yes. The dataset consists of PII in the form of people’s
video, faces, and medical discussions. [Paper section reference: ISS
Dataset/Data Storage and Access]

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? Unknown.

Does the dataset relate to people? Yes, it consists of 29 cancer patients
undergoing active treatment. [Paper section reference: ISS Dataset/Sample and
Study Design]

Does the dataset identify any subpopulations (e.g., by age, sex)? The dataset
is grouped into cohorts assigned by age and sex. [Paper section reference: ISS
Dataset/Sample and Study Design]

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? No. An anonymous identifier has been used to label each patient. The
dataset has been deidentified of patient name, date of birth, address, or
medical record number. [Paper section reference: ISS Dataset/Data Storage and
Access; Appendix/Author Statement]

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? No.

Any other comments? N/A

###  H.3 Collection Process

How was the data associated with each instance acquired? Each patient submits
a video through a mobile or web-based application. [Paper section reference:
ISS Dataset/Patient Protocol]

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)? A
mobile/web-based application is the main interface used to collect the self-
reported pain scores and video recording. Each patient uses their own
smartphone or computer in their personal home setting to submit the video.
[Paper section reference: ISS Dataset/Patient Protocol]

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?
N/A.

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)? Compensation for patients enrolled in the study is explained in the
Consent Form as follows ([Paper section reference: Appendix/Consent Form]):

  * •

$15 per week in which you successfully submit three (3) or more completed
questionnaires, including self-video recordings. A maximum of one (1)
questionnaire may be completed per day.

  * •

$15 per in-clinic questionnaire and video recording session successfully
completed. A minimum of one in-clinic recording must be completed within three
months.

  * •

No matter how many questionnaires or in-clinic sessions you complete, the
total maximum allowed compensation for this study is $225.

Over what timeframe was the data collected? Patients reported in this dataset
were enrolled in the study between December 2020 and July 2021. [Paper section
reference: ISS Dataset/Sample and Study Design; Related Works Table 1]

Were any ethical review processes conducted (e.g., by an institutional review
board)? Yes, NCI protocol 20C0130 entitled, “A Feasibility Study Investigating
the Use of Machine Learning to Analyze Facial Imaging, Voice and Spoken
Language for the Capture and Classification of Cancer/Tumor Pain.”
https://clinicaltrials.gov/ct2/show/NCT04442425 [Paper section reference:
Introduction]

Does the dataset relate to people? Yes.

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)? From the individuals
directly. [Paper section reference: ISS Dataset/Patient Protocol]

Were the individuals in question notified about the data collection? Yes.
([Paper section reference: Appendix/Consent Form])

Did the individuals in question consent to the collection and use of their
data? Yes. [Paper section reference: Appendix/Consent Form]

If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? Yes. Each
patient signed an NIH Assent to Participate in a Clinical Research Study,
reviewed alongside their physician which provides information about the study
such as its purpose, requirements, benefits, and monetary incentives. The form
indicates that the patient may change their mind at any point in time and drop
out of the study. [Paper section reference: ISS Dataset/Sample and Study
Design; Appendix/Consent Form]

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted? Yes, during
the IRB approval process.

Any other comments? N/A

###  H.4 Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? Yes. Video data was
extracted for video frames at 10 frames per second, faces cropped using a
facial detection algorithm, and landmarks (AAMs) detected. Video data was
extracted form audio, and the resulting audio file was used to generate a Mel
spectrogram and audio features. Minimal transformations were applied to the
extracted data. [Paper section reference: ISS Dataset/Data Description;
Appendix/Preprocessing]

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? Yes.

Is the software used to preprocess/clean/label the instances available? Yes,
we use open source Python libraries to include OpenCV for image processing,
ffmpeg for video extraction, PyDub for audio file chunking, Librosa for audio
feature extraction and Mel spectrogram generation, Scikit-Learn for machine
learning models, and PyTorch for neural networks. [Paper section reference:
ISS Dataset/Data Description; Appendix/Preprocessing]

Any other comments? N/A

###  H.5 Uses

Has the dataset been used for any tasks already? No.

Is there a repository that links to any or all papers or systems that use the
dataset? No.

What (other) tasks could the dataset be used for? Our baseline experiments
only addressed one task of static frame-by-frame pain classification. However,
there are a variety of tasks specific to improving self-reported chronic pain
prediction, that can be explored with the ISS dataset. They are:

  * •

Temporal-based, video classification model, integrating image, audio, and
text.

  * •

Disentangling the model’s dependency on patients and their identity.

  * •

Predicting multiple features of pain such as the average self-reported
affective and activity scores.

  * •

Development and automation of chronic cancer pain-specific facial action
units.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? No.

Are there tasks for which the dataset should not be used? General emotion
detection, facial recognition, surveillance applications.

Any other comments? N/A

###  H.6 Distribution

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created? If so, please provide a description. No. NIH will maintain
guardianship of the data.

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identifier (DOI)? To be
determined.

When will the dataset be distributed? To be determined, under the guidance of
the NIH.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions. To be determined, under the
guidance of the NIH. [Paper section reference: Appendix/Author Statement]

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions. No.

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation. No.

Any other comments? Prior to accessing the data, it is likely that users will
be required to provide proof of successful completion of some or all of the
following types of training: Protection of Human Subjects in Research, Global
Privacy and Data Protection, a HIPAA and Health Privacy, General Information
Security, Social and Behavioral Research, and/or Conflicts of Interest.

###  H.7 Maintenance

Who will be supporting/hosting/maintaining the dataset? The NIH. [Paper
section reference: ISS Dataset/Data Storage and Access]

How can the owner/curator/manager of the dataset be contacted (e.g., email
address<EMAIL_ADDRESS>
Is there an erratum? If so, please provide a link or other access point. N/A.

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)? To be
determined, under the guidance of the NIH.

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were individuals in question
told that their data would be retained for a fixed period of time and then
deleted)? If so, please describe these limits and explain how they will be
enforced. Patients were informed through the Consent Form that their data may
be held indefinitely. [Paper section reference: Appendix/Consent Form]

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to users. To be determined, under the guidance of the NIH.

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
other users? If so, please provide a description. To be determined, under the
purview of the NIH.

Any other comments? N/A.

##  Appendix I Risk Categorization

Our study was classified as “Minimal Risk under 45 CFR 46 / 21 CFR 56” after
review and approval by the National Institutes of Health’s Federalwide
Assurance (FWA00005897), in accordance with Federal regulations 45 CFR 46 and
21 CFR 56, including Good Clinical Practice. The study also was determined to
fall under three “Expedited Review Categories” due to three applicable
characteristics including (1b) clinical studies of medical devices for which
an investigational device exemption application (21 CFR Part 812) is not
required, (6) collection of data from voice, video, digital, or image
recording made for research purposes, and (7) research on individual or group
characteristics or behavior. The data collection which involved the use of a
mobile or web app to input answers to questions and record videos do not pose
risk to patients. The tasks also do not involve deception.