# A Benchmark of Domain-Adapted Large Language Models for Generating Brief
Hospital Course Summaries

Asad Aali  Department of Electrical and Computer Engineering, The University
of Texas at Austin, Austin, TX, USA  Dave Van Veen  Department of Electrical
Engineering, Stanford University, Stanford, CA, USA  Center for Artificial
Intelligence in Medicine and Imaging, Palo Alto, CA, USA  Yamin Ishraq Arefeen
Department of Electrical and Computer Engineering, The University of Texas at
Austin, Austin, TX, USA  Jason Hom  Department of Medicine, Stanford, CA, USA
Christian Bluethgen  Department of Medicine, Stanford, CA, USA  University
Hospital Zurich, Zurich, Switzerland  Eduardo Pontes Reis  Center for
Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA  Albert
Einstein Israelite Hospital, São Paulo, Brazil  Sergios Gatidis  Department of
Radiology, Stanford University, Stanford, CA, USA  Center for Artificial
Intelligence in Medicine and Imaging, Palo Alto, CA, USA  Namuun Clifford
School of Nursing, The University of Texas at Austin, Austin, TX, USA  Joseph
Daws  One Medical, San Francisco, CA, USA  Arash S. Tehrani  One Medical, San
Francisco, CA, USA  Jangwon Kim  Amazon, Seattle, WA, USA  Akshay S. Chaudhari
Department of Radiology, Stanford University, Stanford, CA, USA  Department of
Biomedical Data Science, Stanford University, Stanford, CA, USA  Center for
Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA

###### Abstract

Background: Brief hospital course (BHC) summaries are common clinical
documents generated by summarizing clinical notes. While large language models
(LLMs) depict remarkable capabilities in automating real-world tasks, their
capabilities for healthcare applications such as BHC synthesis have not been
shown. To enable the adaptation of LLMs for BHC synthesis, we introduce a
novel benchmark consisting of a pre-processed dataset extracted from MIMIC-IV
notes, encapsulating clinical note, and brief hospital course (BHC) pairs. As
part of the benchmark, we assess the performance of two general-purpose LLMs
and three healthcare-adapted LLMs to improve BHC synthesis from clinical
notes.

Methodology: Using clinical notes as input for generating BHCs, we apply
prompting-based (using in-context learning) and fine-tuning-based adaptation
strategies to three open-source LLMs (Clinical-T5-Large, Llama2-13B, FLAN-UL2)
and two proprietary LLMs (GPT-3.5, GPT-4). We quantitatively evaluate the
performance of these LLMs across varying context-length inputs using
conventional natural language similarity metrics. We further perform a
qualitative study where five diverse clinicians blindly compare clinician-
written BHCs and two LLM-generated BHCs for 30 samples across metrics of
comprehensiveness, conciseness, factual correctness, and fluency. We compare
reader preferences for the original and LLM-generated summary using Wilcoxon
Signed-Rank tests.

Results: The Llama2-13B fine-tuned LLM outperformed other domain-adapted
models given quantitative evaluation metrics of BLEU and BERT-Score. GPT-4
with prompting adaptation showed more robustness to increasing context lengths
of clinical note inputs than fine-tuned Llama2-13B. The reader study depicted
a significant preference for summaries generated by GPT-4 with in-context
learning compared to both Llama2-13B fine-tuned summaries and the original
summaries ($p<0.001$).

Conclusion: We present a new benchmark and pre-processed dataset for using
LLMs in BHC synthesis from clinical notes. We observe high-quality
summarization performance for both in-context proprietary and fine-tuned open-
source LLMs using both quantitative metrics and a qualitative clinical reader
study. We propose our work as a benchmark to motivate future works to adapt
and assess the performance of LLMs in BHC synthesis.

Table 1: a) A sample of our novel pre-processed clinical notes dataset,
extracted from raw MIMIC-IV notes.

Input | Example
---|---
SEX | F
SERVICE | SURGERY
ALLERGIES | No Known Allergies
CHIEF COMPLAINT | Splenic laceration
MAJOR PROCEDURE | NONE
HISTORY OF PRESENT ILLNESS | s/p routine colonoscopy this morning with polypectomy (report not available) …
PAST MEDICAL HISTORY | Mild asthma, hypothyroid
FAMILY HISTORY | Non-contributory
PHYSICAL EXAM | Gen: Awake and alert CV: RRR Lungs: CTAB Abd: Soft, nontender, nondistended
PERTINENT RESULTS | 03:45 PM BLOOD WBC-5.5 RBC-3.95 Hgb-14.1 …
MEDICATIONS ON ADMISSION | 1\. Levothyroxine Sodium 100 mcg PO DAILY 2. Flovent HFA (fluticasone) …
DISCHARGE DISPOSITION | Home
DISCHARGE DIAGNOSIS | Splenic laceration
DISCHARGE CONDITION | Mental Status: Clear and coherent. Level of Consciousness: Alert and interactive …
DISCHARGE INSTRUCTIONS | You were admitted to … in the intensive care unit for monitoring after a …
Output | Example
BRIEF HOSPITAL COURSE | Ms. … was admitted to … on …. After getting a colonoscopy and polypectomy, she …

b) Relevant statistics for the pre-processed dataset split across multiple
context length ranges for adaptation tasks.

Context Range Samples Input Tokens BHC Tokens $0$ \- $1,024$ $2,000$ $711\pm
199$ $104\pm 43$ $1,024$ \- $2,048$ $2,000$ $1,471\pm 275$ $148\pm 36$ $2,048$
\- $4,096$ $2,000$ $2,496\pm 388$ $225\pm 55$

## Introduction

Clinicians spend significant time with clinical documentation activities,
which are time-consuming and reduce the time that can be spent on patient-
facing activities [1, 2, 3]. Brief hospital course (BHC) summaries are
encapsulated within clinical notes, which are commonly written for
hospitalization cases. Synthesizing BHCs from multiple clinical notes requires
substantial clinician time and expertise, with errors possibly causing patient
harm [4]. Automating or accelerating BHC note generation is important since
discharge summaries can either lack essential information or contain incorrect
information, and may be incomplete before follow-up appointments [5]. The
recent success of generative large language models (LLMs) has provided a
promising avenue for this clinically important task of BHC synthesis from
clinical notes.

In this study, we present a novel benchmark111We will be releasing the code
for data preprocessing, model adaptation, and evaluation upon publication. and
dataset222Our preprocessed clinical notes dataset will be made publicly
available at the time of publication. curated from MIMIC-IV notes [6] for the
task of BHC summarization. We benchmark the performance of domain-adapted LLMs
in the BHC summarization task to motivate future benchmarking studies. We
compare several state-of-the-art (SOTA) LLMs: proprietary general-purpose
LLMs, such as GPT-3.5 [7] and GPT-4 [8, 9], open-source LLMs, such as Llama2
[10], FLAN- UL2 [11], and a T-5 model pre-trained on medical text [12]. We
explore prompting-based (using in-context learning [ICL]) and fine-tuning-
based adaptation strategies for the proprietary and open-source LLMs used in
our benchmarking study to identify the best model and adaptation strategy. We
further benchmark and provide a comprehensive quantitative analysis for BHC
synthesis using well-established metrics measuring the syntactic and semantic
similarity between LLM-generated and clinician-written BHCs. With such
metrics, we further explore the robustness of domain-adapted LLMs across
clinical notes with higher context lengths. Finally, we conduct a rigorous
clinical reader study involving five expert clinicians reviewing clinical
notes and BHC pairs to ensure adequate qualitative clinical evaluation of the
LLM-generated summaries across dimensions of comprehensiveness, conciseness,
factual correctness, and fluency. Our novel benchmark which includes the
preprocessed dataset, LLM adaptation, quantitative and clinical evaluation,
seeks to improve the possibilities of using LLMs for clinical text
summarization tasks.

## Related Works

Recent advancements in natural language processing (NLP) have been
characterized by the adoption of transformer-based language models [13].
Transformer models such as Bidirectional Encoder Representations from
Transformers (BERT) [14] and Unidirectional Generative Pre-trained Transformer
2 (GPT-2) [15] paved the way for training large-scale autoregressive models.
The evolution of transformer models involved pretraining on large corpora of
text data and subsequent fine-tuning on domain-specific data, as demonstrated
by GPT-3 [16], PaLM [17], and T5 [18]. After extensive pre-training, these
LLMs are often optimized for alignment with human preferences, typically using
instruction tuning [19]. Prior works have demonstrated successful adaptation
of LLMs in the medical domain by pretraining or fine-tuning LLMs on clinical
text [20].

Figure 1: Overall schematic of our study. We evaluate a variety of models,
including open-source models containing up to 20 billion parameters, and
larger-scale proprietary models. Each model is adapted to the summarization
task using the adaptation strategies displayed (except QLoRA is not applied to
GPT-3.5 and GPT-4 since these proprietary models do not provide access to
model weights). We evaluate each model’s performance by comparing its outputs
with expert clinician summaries. Each model paired with the adaptation
strategy is evaluated using quantitative similarity metrics. Finally, we
perform a clinical study where five clinicians rate three summaries
(randomized order) for every summarization task: best-performing open-source
model, best-performing proprietary model, and clinician-written. The ∗
indicates GPT-4’s maximum context length at the time of experimentation.
Later, the maximum context length was increased to 128,000.

## Methods

### Dataset

For our task of synthesizing BHCs from clinical notes, we utilized the raw
MIMIC-IV-Note dataset [6], a compilation of 331,794 de-identified discharge
summaries from 145,915 patients admitted to the Beth Israel Deaconess Medical
Center for a diverse range of medical scenarios. We pass MIMIC-IV [6] notes
through our pre-processing pipeline (described below) to extract a novel
dataset containing 270,033 clinical notes and BHC pairs. This dataset is
tailored for the task of BHC summarization with a focus on the relationship
between clinical notes and corresponding BHCs. Creating this benchmark dataset
is crucial for others to replicate findings and to compare future works.

### Data Processing

Our pipeline uses regular expressions to remove special characters, and
unnecessary text, and to separate each section of the clinical note using line
breaks to create appropriate section headings. We utilize pre-trained
tokenizers from HuggingFace to further process the dataset for LLM
training/inference. We extract the BHCs from each clinical note to create a
labeled dataset with relevant input-output, clinical notes–BHC pairs. Our pre-
processing results generate 270,033 samples with an average input token length
of 2,267 $\pm$ 914 and an average output token length of 564 $\pm$ 410\. We
further create three independent train and test datasets for our adaptation
tasks through random sampling as described in Table
LABEL:table:dataset_sample. To allow for an unbiased analysis, we also
randomly sample 100 independent test examples from the pre-processed dataset
for each context length range. We use the smallest context length dataset (0 -
1,024 tokens) for the initial model evaluation to make extensive quantitative
analysis feasible.

### Large Language Models

We categorize the LLMs used in this study as open-source LLMs and proprietary
LLMs.

#### Open-Source LLMs

The models in this category contain up to 20 billion parameters and can be
fine-tuned on consumer-grade GPUs. These models were trained either with a
sequence-to-sequence (seq2seq) or autoregressive objective. The seq2seq
architecture maps input sequences directly to output sequences. In contrast,
autoregressive architectures predict the next token given the preceding
context. The following models were considered:

  * •

Clinical-T5-Large is a "text-to-text transfer transformer" [12] pre-trained on
medical text, that employs transfer learning with the Sequence-to-Sequence
(seq2seq) architecture to clinical tasks. Clinical-T5-Large supports only a
limited context window of 512 tokens.

  * •

FLAN-UL2 hails from the T5 [18] family, utilizing the seq2seq architecture. It
employs a modified pre-training procedure to incorporate an expanded context
length of 2,048 tokens instead of only 512 tokens context length of the
original FLAN-T5. FLAN-UL2 training also involved additional instruction
tuning [21] to enhance the model’s capability to understand complex
narratives.

  * •

Llama2-13B stems from the Llama family of LLMs [10] and is an open-source
autoregressive model tailored for expanded pretraining on 2 trillion tokens.
Llama2-13B allows up to 4,096 tokens as input.

#### Proprietary LLMs

This category includes large-scale proprietary autoregressive models that
utilize reinforcement learning with human feedback (RLHF) to further improve
performance over instruction tuning using feedback from expert human
evaluators.

  * •

ChatGPT (GPT-3.5) [7] contains 175 billion parameters and has been extensively
fine-tuned for general tasks using human feedback, enhancing its ability to
capture intricate details within any summarization task. GPT-3.5 can support
input context lengths of up to 16,384 tokens.

  * •

GPT-4 [9] achieves state-of-the-art overall NLP performance and supports an
input context window of 128,000 tokens, which is the highest compared to other
models listed in Figure 1, making it a suitable choice in multi-document
summarization scenarios.

Clinical-T5-Large [12], and FLAN-UL2 [11] have a maximum input context length
limit of 512 and 2,048 tokens, respectively. Hence, for the context length
analysis in our results section, we only select models that allow at least
4,096 context length inputs, allowing exploration of model behavior for
varying extents of the input length. We used the GPT-3.5 and GPT-4 API for all
our summarization tasks.

### Adaptation Strategies

For each pre-trained model outlined in the previous section, we apply a series
of lightweight domain adaptation methods as shown in Figure 1. We only explore
adaptation through discrete prompting (no fine-tuning) for large-scale
proprietary LLMs since the model weights are not accessible. The adaptation
methods mentioned below gradually increase the level of adaptation to a
downstream task.

  * •

Null Prompting: This is a discrete zero-shot adaptation strategy [22] where
the clinical note is supplied with a basic prompt, such as "brief hospital
course". This approach is a baseline technique to evaluate how models respond
to minimal task-specific guidance.

  * •

Prefix Prompting: Building upon the null prompting approach, we now provide a
more detailed instructional prompt [22, 21], like "summarize the following
clinical note," which serves to provide the model with additional context for
the BHC synthesis task.

  * •

In-Context Prompting: We employ in-context learning (ICL) [21] using a
discrete few-shot prompt. ICL involves providing the LLM with examples and
instructions within the prompt itself, demonstrating a desired behavior [21].
We enhance our null prompting strategy by prepending one example clinical note
and BHC pair, selected randomly from the training dataset.

  * •

Quantized Low-Rank Adaptation (QLoRA)[23, 24]: This technique utilizes 4-bit
quantization to enable parameter efficient fine-tuning[25] by injecting rank-
decomposition matrices into each layer of the model. We use QLoRA to fine-tune
LLMs on the supervised task of BHC synthesis using our curated dataset. QLoRA
fine-tuning was performed on a mix of NVIDIA A10G and V100 GPU machines.

Our adaptation methods are carefully chosen to strike a balance between
computational efficiency and task-specific adaptability. Null prompting and
prefixed prompting serve as foundational approaches, allowing us to gauge
model responsiveness with minimal and augmented task-specific cues. In-context
learning further refines adaptation by introducing contextually relevant
examples. QLoRA provides an efficient means of fine-tuning [25] larger models,
which is crucial for handling the complexity of the BHC summarization task.

### Experimental Setup

#### Quantitative Evaluation

To identify the best-performing open-source and proprietary models, we
conducted a performance analysis, where we selected a subset of 70 independent
samples from the 100 test samples in the 0-1,024 context range. The remaining
30 independent samples from this range were separated for a reader study
described below, to avoid data leakage. To accurately assess the performance
of our BHC summarization models, we employ quantitative metrics commonly used
in summarization tasks to evaluate the syntactic similarity and semantic
relevance of the generated BHC summaries. Bilingual Evaluation Understudy
(BLEU) [26] evaluates the overlap between the generated and reference
summaries using a weighted average of 1 to 4-gram precision. ROUGE-L [27]
assesses the precision and recall of the longest common subsequence. BLEU and
ROUGE-L depict lexical overlap between generated and reference summaries. In
contrast, we also use semantic similarity metrics like BERT-Score [28] which
leverages contextual BERT embeddings to evaluate the similarity between the
generated and reference summaries. Finally, we perform an independent context-
length quantitative analysis using fine-tuned Llama2-13B and GPT-4 with ICL,
the best-performing open-source and proprietary LLMs allowing context lengths
greater than 4,000 tokens as input using all 100 test samples from each of the
three context-length test sets. To assess potential health equity
implications, we perform a stratified subgroup analysis across patient-
reported sex when assessing the quantitative performance of BHC summarization
techniques.

Figure 2: Quantitative metric results for each choice of model, across
increasing domain-adaptation strategies. In summary, QLoRA as an adaptation
strategy outperforms other adaptation methods. Specifically, QLoRA Llama2-13B
outperforms other models in BLEU score, while achieving comparable performance
to Clinical-T5-Large in BERT-Score and ROUGE-L.

#### Qualitative Evaluation

We conducted a reader study with five board-certified clinicians to compare
BHCs written by expert clinicians during clinical practice from the MIMIC
dataset to BHCs generated by our best-performing open-source and proprietary
models. We utilized 30 independent clinical notes and BHC pairs from the 0 -
1,024 context-length dataset. Each of the 30 independent clinical notes were
paired with three BHCs: (1) original BHC written by an expert clinician, (2)
BHC generated by the best-performing open-source LLM, and (3) BHC generated by
the best-performing proprietary LLM. These were then presented to a group of
five diverse clinicians from multiple institutions and different backgrounds,
each with a clinical focus in internal medicine, nursing, thoracic radiology,
pediatric radiology, and neuroradiology (with 11, 7, 5, 6, and 7 years of
experience, respectively). We chose diverse clinical expertise and
institutions to reflect the diversity in clinical practice. The BHCs were
presented in randomized and blinded order to the clinicians. The five
clinicians ranked the 30 note-summary pairs on a Likert scale of 1 (poor) to 5
(excellent) based on four evaluation criteria:

  * •

Comprehensiveness: How well does the summary capture important information?
This assesses the recall of clinically significant details from the input
text.

  * •

Conciseness: How well does the summary exclude non-important information? This
compares how well the summary is condensed, considering the value of a summary
decreases with superfluous information.

  * •

Factual Correctness: How well does the summary agree with the facts outlined
in the clinical note? This evaluates the precision of the information
provided.

  * •

Fluency: How well does the summary exhibit fluency? This assesses the
readability and natural flow of the content.

### Statistical Analysis

We explored the following two hypotheses: 1) Does our pool of five diverse
clinicians exhibit a preference for LLM-generated summaries over clinician-
written ones?, and 2) Does our pool of five diverse clinicians exhibit a
preference for summaries generated by adapted proprietary LLMs over summaries
generated by adapted open-source LLMs?. We perform these tests across each of
the four reader study evaluation criteria, comparing the best-performing
proprietary and open-source LLMs with clinician summaries. We performed
significance testing using non-parametric Wilcoxon signed-rank tests, with a
significance level $\alpha$ $=0.05$, and with Bonferroni corrections using
XLSTAT.

Figure 3: a) Quantitative evaluation metrics across increasing input context
lengths. GPT-4 shows consistency in performance whereas Llama2-13B shows a
drop in summarization with increasing context length inputs. b) Context size
analysis for QLoRA Llama2-13B (in/out-of-distribution), where each item on the
y-axis displays an independent model fine-tuned on samples from a specific
context length range. The summarization performance of the combined model
trained on 0 - 4,096 context length inputs outperforms other models with
longer input clinical notes at inference (more than 1,024 tokens).

## Results

### Model Performance Analysis

Despite being the smallest model, Clinical-T5-Large (750M parameters),
exhibited competitive performance after QLoRA fine-tuning and achieved the
largest performance improvement with increasing adaptation (Figure 2). FLAN-
UL2 displayed strong performance across all adaptation strategies, except ICL.
Its improvement in performance with fine-tuning was limited. Llama2-13B
performance improved substantially following QLoRA fine-tuning, achieving the
highest BLEU and BERT scores across all adaptation strategies and LLMs.
GPT-3.5 showed good overall performance but exhibited limited benefits with
increasing domain adaptation. GPT-4 with ICL outperformed GPT-3.5 and became
the top-performing proprietary LLM. When evaluating the deviation of each
model’s overall summarization performance across the subgroups of patient-
reported sex, we noticed large variations with the Clinical-T5-Large model
(Figure 2). With increasing adaptation from null prompting to QLoRA, GPT
models showed higher variance from their original ROUGE-L and BERT scores.
LLMs after QLoRA adaptation generally showed lower variations among subgroups
than other adaptation strategies.

### Context Length Analysis

We compared the performance of our LLMs when the adaptation (training) and
inference (testing) context lengths were identical lengths (in-distribution)
(Figure 3 a)). QLoRA Llama2-13B sees a drop in in-distribution performance. In
contrast, in-context GPT-4 displays consistent performance in-distribution,
across all metrics as the context length of the train and test set increases.
Overall, in-context GPT-4 exhibits more robustness than QLoRA Llama2-13b as
context lengths increase. We further explored LLM performance with differing
training and testing context lengths (Figure 3 b)). We observed that adapted
models at testing perform best on samples with context lengths similar to what
they were trained on (in-distribution), independent of the length of the
samples. Subsequently, the performance of models trained on smaller context-
length samples deteriorates when testing on longer-context-length samples.

### Clinical Reader Study

Figure 4: Violin plot showing results from the reader study with five
clinicians. Clinicians exhibit a clear and strong preference for in-context
GPT-4 (adapted large-scale proprietary LLM) summaries over QLoRA Llama2-13B
(adapted open-source LLM) and clinician-written summaries with statistical
significance by the Wilcoxon signed-rank test, achieving *$p<0.001$ across
each attribute. (NS: Not Significant).

We find that our five clinical readers strongly prefer in-context GPT-4
summaries for attributes of comprehensiveness, conciseness, factual
correctness, and fluency, compared to the original clinician-written summaries
($p<0.001$) and QLoRA Llama2-13B summaries ($p<0.001$) (Figure 4). However, we
find no significant differences between reader preferences comparing Llama2
summaries and the original clinician-written summaries ($p=0.05$). Overall,
GPT-4 scores have a narrower range, showcasing lower variation. We also report
the proportion of scores from readers between the range of 3 to 5 for GPT-4,
Llama2-13B, and clinician summaries: 100%, 47%, and 40%, respectively. This
study underscores the strength of LLMs in generating summaries that are both
statistically superior and preferred by our panel of clinicians in the
majority of cases.

## Discussion

In this study, we present a new open-source benchmark for generating BHC
summaries from multiple clinical notes using domain-adapted open-source and
closed-source LLMs, coupled with quantitative and qualitative evaluation. Our
benchmark shows that open-source LLMs can produce high-quality summaries and
that Llama2-13B comes out as a superior choice for further analysis of open-
source LLMs based on its performance and an expanded context length window.
The subgroup analysis further emphasizes the importance of a multi-faceted
analysis when evaluating models. Furthermore, our context-length analysis
showcases the benefit of fine-tuning LLMs on datasets with a wide range of
context lengths, especially when summarizing longer clinical notes.

Our evaluation of LLM summaries suggests that adapted open-source LLMs like
Llama2-13B can produce summaries similar in quality to clinician summaries as
depicted in reader preferences (Figure 4). Open-source models also provide the
benefit of ease in implementation, where sending data via an API is not
possible. These models can be fine-tuned and implemented on-premise, as their
weights are publicly available. Our benchmark also reports strong
summarization performance from Clinical-T5-Large, while it is the smallest
parameter model. A possible explanation of this could be extensive pre-
training on medical text. However, Clinical-T5-Large also presented a large
variation in performance across subgroups, making it a less suitable choice
for deployment.

In our study, we combine evaluation via quantitative metrics and qualitative
assessments to capture a comprehensive overview of the performance of
different summarization methods. It is crucial to recognize that quantitative
measures, such as NLP metrics, and qualitative assessments by clinical readers
gauge different aspects of summarization quality. NLP metrics, for instance,
quantify the level of (textual, semantic, contextual) similarity between two
summaries, whereas clinical readers inform perceptual (subjective) quality
differences. We observed that fine-tuned Llama2-13B consistently outperformed
in-context GPT-4 in quantitative similarity metrics, suggesting a closer
resemblance to clinician-written summaries. However, the reader study results
present a nuanced narrative where clinicians express a preference for
GPT-4-generated summaries. This finding eludes a critical point: high
quantitative similarity scores do not necessarily translate to a "better
summary" from a clinician’s perspective, as well as across subgroups. Hence,
we advocate for the critical role of human evaluation. Enhancing the alignment
between quantitative metrics and qualitative assessments will enable a more
holistic evaluation for clinical text summarization[31].

Recent adaptations of LLMs in the clinical domain emphasize their potential in
understanding medical language. Noteworthy recent works [30, 31, 29] have used
open-source datasets and displayed the performance of LLMs in multiple
summarization tasks in medicine: radiology reports, patient health questions,
progress notes, and doctor-patient conversations. Some prior works have
further explored the use of proprietary models like ChatGPT [7] in radiology
report summarization [32], as well as discharge summarization [33, 34]. A
recent work presents a large-scale multi-document dataset for brief hospital
courses, promoting the advancement of hospital visit summarization [35].
Another work [36] further explores domain adaptation of LLMs through fine-
tuning for improved summarization of clinical notes. Models like Clinical-T5
[12] and Med-PaLM [37] have shown promise in the clinical domain for tasks
such as named entity recognition, natural language inference, and question-
answering. Our study takes inspiration from related works to develop a novel
benchmark using our preprocessed dataset, presenting a comprehensive
evaluation of adapted LLMs. Through rigorous quantitative, qualitative, and
subgroup evaluation, we ultimately aim to bridge the gap between machine
learning research and clinical adoption.

While noting the strengths of our study, we also discuss its limitations. Our
selection of LLMs for the BHC adaptation task is not comprehensive. As new
adaptation strategies are introduced, and new LLMs are trained and released, a
promising direction for future works would be to evaluate their performance on
our preprocessed dataset and compare it with our proposed benchmark. Another
limitation of our work is the limited availability of publicly available
datasets for clinical note summarization, making it difficult to gauge the
performance of our adapted LLMs in diverse BHC synthesis datasets across
hospitals and clinical practices.

## Conclusion

In this investigation, we developed a benchmarking dataset and performed a
comprehensive quantitative and qualitative evaluation of using LLMs for
synthesizing BHCs from clinical notes using adaptation strategies for both
open-sourced and closed-sourced models. Via a clinical reader study, we
depicted that adapted open-source models can match the quality of clinician-
generated summaries, while adapted proprietary models can outperform the
quality of clinician-generated summaries across dimensions of
comprehensiveness, conciseness, factual correctness, and fluency. Our study
can help define a framework for a technical and clinical evaluation of LLMs
and our findings have the potential to alleviate the documentation burden on
clinicians.

## Acknowledgments

Computing resources were partially provided by One Medical and Stanford
University. Microsoft provided Azure OpenAI credits for this project via the
Accelerate Foundation Models Academic Research (AFMAR) program. A.C. receives
research support from NIH grants R01 HL167974, R01 AR077604, R01 EB002524, R01
AR079431, and P41 EB027060; from NIH contracts 75N92020C00008 and
75N92020C00021; from Stanford Center for Artificial Intelligence and Medicine,
Stanford Institute for Human Centered AI, from Stanford Center for Digital
Health, from Stanford Cardiovascular Institute, from Stanford Center for
Precision Health and Integrated Diagnostics; from GE Healthcare, Philips and
Amazon.

## References

  * [1] Moy, A., Schwartz, J., Chen, R., et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. Journal Of The American Medical Informatics Association. 28, 998-1008 (2021)
  * [2] Chaiyachati, K., Shea, J., Asch, D., et al. Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations. JAMA Internal Medicine. 179, 760-767 (2019)
  * [3] Mamykina, L., Vawdrey, D. & Hripcsak, G. How do residents spend their shift time? A time and motion study with a particular focus on the use of computers. Academic Medicine: Journal Of The Association Of American Medical Colleges. 91, 827 (2016)
  * [4] Clough, R., Sparkes, W., Clough, O., Sykes, J., Steventon, A. & King, K. Transforming healthcare documentation: Harnessing the potential of AI to generate discharge summaries. BJGP Open. (2023)
  * [5] Kripalani, S., LeFevre, F., Phillips, C., Williams, M., Basaviah, P. & Baker, D. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. Jama. 297, 831-841 (2007)
  * [6] Johnson, A., Pollard, T., Horng, S., Celi, L. & Mark, R. MIMIC-IV-Note: Deidentified free-text clinical notes. (PhysioNet,2023)
  * [7] OpenAI OpenAI. ChatGPT. Accessed September 10, 2023. [Online]. (2022), https://openai.com/blog/chatgpt
  * [8] Bubeck, S., Chandrasekaran, V., Eldan, R., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv Preprint ArXiv:2303.12712. (2023)
  * [9] OpenAI GPT-4 Technical Report. (2023)
  * [10] Touvron, H., Lavril, T., Izacard, G., et al. Llama: Open and efficient foundation language models. ArXiv Preprint ArXiv:2302.13971. (2023)
  * [11] Tay, Y., Dehghani, M., Tran, V., et al. Unifying language learning paradigms. ArXiv Preprint ArXiv:2205.05131. (2022)
  * [12] Lehman, E. & Johnson, A. Clinical-t5: Large language models built using mimic clinical text. (PhysioNet,2023)
  * [13] Vaswani, A., Shazeer, N., Parmar, N., et al. Attention is All you Need. Advances In Neural Information Processing Systems. 30 (2017), https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [14] Kenton, J. & Toutanova, L. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings Of NaacL-HLT. 1 pp. 2 (2019)
  * [15] Radford, A., Wu, J., Child, R., et al. Language models are unsupervised multitask learners. OpenAI Blog. 1, 9 (2019)
  * [16] Brown, T., Mann, B., Ryder, N., et al. Language Models are Few-Shot Learners. Advances In Neural Information Processing Systems. 33 pp. 1877-1901 (2020), https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * [17] Chowdhery, A., Narang, S., Devlin, J., et al. Palm: Scaling language modeling with pathways. Journal Of Machine Learning Research. 24, 1-113 (2023)
  * [18] Raffel, C., Shazeer, N., Roberts, A., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal Of Machine Learning Research. 21, 5485-5551 (2020)
  * [19] Zhang, S., Dong, L., Li, X., et al. Instruction Tuning for Large Language Models: A Survey. (2023)
  * [20] Wang, B., Xie, Q., Pei, J., et al. Pre-trained language models in biomedical domain: A systematic survey. ACM Computing Surveys. 56, 1-52 (2023)
  * [21] Lampinen, A., Dasgupta, I., Chan, S., et al. Can language models learn from explanations in context?. ArXiv Preprint ArXiv:2204.02329. (2022)
  * [22] Wei, J., Wang, X., Schuurmans, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances In Neural Information Processing Systems. 35 pp. 24824-24837 (2022)
  * [23] Hu, E., Shen, Y., Wallis, P., et al. Lora: Low-rank adaptation of large language models. ArXiv Preprint ArXiv:2106.09685. (2021)
  * [24] Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. ArXiv Preprint ArXiv:2305.14314. (2023)
  * [25] Ding, N., Qin, Y., Yang, G., et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence. 5, 220-235 (2023)
  * [26] Papineni, K., Roukos, S., Ward, T. & Zhu, W. Bleu: a method for automatic evaluation of machine translation. Proceedings Of The 40th Annual Meeting Of The Association For Computational Linguistics. pp. 311-318 (2002)
  * [27] Lin, C. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. pp. 74-81 (2004)
  * [28] Zhang, T., Kishore, V., Wu, F., Weinberger, K. & Artzi, Y. Bertscore: Evaluating text generation with bert. ArXiv Preprint ArXiv:1904.09675. (2019)
  * [29] Chen, Z., Varma, M., Wan, X., Langlotz, C. & Delbrouck, J. Toward expanding the scope of radiology report summarization to multiple anatomies and modalities. ArXiv Preprint ArXiv:2211.08584. (2022)
  * [30] Van Veen, D., Van Uden, C., Attias, M., et al. RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models. ArXiv Preprint ArXiv:2305.01146. (2023)
  * [31] Van Veen, D., Van Uden, C., Blankemeier, L., et al. Clinical text summarization: adapting large language models can outperform human experts. ArXiv Preprint ArXiv:2309.07430. (2023)
  * [32] Lyu, Q., Tan, J., Zapadka, M., et al. Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential. Visual Computing For Industry, Biomedicine, And Art. 6, 9 (2023)
  * [33] Patel, S. & Lam, K. ChatGPT: the future of discharge summaries?. The Lancet Digital Health. 5, e107-e108 (2023)
  * [34] Singh, S., Djalilian, A. & Ali, M. ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes. Seminars In Ophthalmology. pp. 1-5 (2023)
  * [35] Adams, G., Alsentzer, E., Ketenci, M.,et al. What’s in a summary? laying the groundwork for advances in hospital-course summarization. Proceedings Of The Conference. Association For Computational Linguistics. North American Chapter. Meeting. 2021 pp. 4794 (2021)
  * [36] Wang, H., Gao, C., Dantona, C., Hull, B. & Sun, J. DRG-LLaMA: Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patients. ArXiv Preprint ArXiv:2309.12625. (2023)
  * [37] Singhal, K., Azizi, S., Tu, T., et al. Large language models encode clinical knowledge. Nature. 620, 172-180 (2023)


# Exploring and Applying Audio-Based
Sentiment Analysis in Music

Etash Jhanji<EMAIL_ADDRESS>
Fox Chapel Area High School, Pittsburgh PA USA

###### Abstract

Sentiment analysis is a continuously explored area of text processing that
deals with the computational analysis of opinions, sentiments, and
subjectivity of text. However, this idea is not limited to text and speech, in
fact, it could be applied to other modalities. In reality, humans do not
express themselves in text as deeply as they do in music. The ability of a
computational model to interpret musical emotions is largely unexplored and
could have implications and uses in therapy and musical queuing. In this
paper, two individual tasks are addressed. This study seeks to (1) predict the
emotion of a musical clip over time and (2) determine the next emotion value
after the music in a time series to ensure seamless transitions. Utilizing
data from the Emotions in Music Database, which contains clips of songs
selected from the Free Music Archive annotated with levels of valence and
arousal as reported on Russel’s circumplex model of affect by multiple
volunteers, models are trained for both tasks. Overall, the performance of
these models reflected that they were able to perform the tasks they were
designed for effectively and accurately.

##  1 Introduction

Although sentiment analysis is well researched, studies focus on text-based
emotion and represent emotions categorically.[6] People often express strong
emotions through speech and audio, with more emphasis than in text. Other
works focus largely on speech sentiment[10], not necessarily incorporating
other sounds in an audio file. A computational model to interpret musical
emotions can have medical uses, including therapeutic purposes, and also
commercial uses in queueing music for streaming services. This project seeks
to explore the capability of Long Short-Term Memory (LSTM) models to (1)
predict the emotion of a musical clip over time and (2) determine the next
"emotion value" after in a time series to ensure seamless transitions between
audio sentiments.

##  2 Background

###  2.1 Representing Emotion

Typically, text-based sentiment analysis attempts to classify text into
categories of basic emotions as is supported by neuroscience studies on
distinct neural circuits for emotions.[13] On the other hand, Russel’s
circumplex model of affect proposes that emotions arise as a product of two
separate circuits: arousal and valence. These can also be renamed as
activation and pleasantness, respectively.[8] This redefines the sentiment
analysis task as a regression task with two outputs rather than a
classification task. The database used also utilizes Russel’s model on audio
samples as the samples are annotated to show the emotion intended to be
induced by the clip. Furthermore, the model can be divided into regions to
represent emotions with more precision and can additionally describe
intensity/severity as can be seen in Fig. 1. This is useful in the
applications of audio analysis and provides a quantitative measure of emotion.

Figure 1: Russel’s Circumplex Model of Affect

###  2.2 Audio Processing

There are many ways to represent audio files for input to a model, but this
study uses Logarithmic Mel Spectrograms. This works by taking the raw audio
waveform, applying a Fourier Transform, creating a spectrogram of the
frequencies, applying the Mel scale to represent human frequencies better, and
applying a logarithmic transformation.[4] Random Gaussian noise is also added
to the Mel Spectrogram to improve the robustness of the model. The audio clips
were spliced into half-second clips, sampled at 44100 Hz, and the Mel
spectrogram has 128 frequency bins, an FFT length of 512, and a hop length of
2048 with Librosa. The output was a NumPy array or PyTorch tensor with a shape
of 128 by 44 for 128 bins and 44 timesteps. Alternatively, the model could
have used Mel Frequency Cepstral Coefficients (MFCCs) which tend to represent
speaker data well[2], however, the model should not have to depend on speech
features and should instead be able to understand purely instrumentals tracks
as well. The extracted Mel Spectrogram dataset was then tied to its continuous
arousal and valence coordinates. The full pipeline is demonstrated in Fig. 2.

Figure 2: The full pipeline of audio processing including clipping, mel
spectrogram, and storage format.

###  2.3 Long Short-Term Memory Models

LSTM models are a subset of recurrent neural networks (RNNs). These models
have a hidden state encoding previous information to keep a stateful memory of
previous entries to the model. Traditional RNNs have a vanishing gradient
problem where previous inputs are forgotten over time, but LSTMs solve this
problem with a more sophisticated architecture. This type of model is able to
outperform traditional models where long-term dependencies play a part in
prediction, such as in audio where data takes place over time.

###  2.4 Dataset [1, 11]

The Emotion in Music Database (EmoMusic/1000 songs) consists of about 700
free-use songs that have been annotated by volunteers. The crowdsourced data
consists of annotations sampled at 2 Hz (every 0.5 seconds) for arousal and
valence of a clip on a scale from -1 to 1 each. The continuously sampled data
points are the average of 10 volunteers’ annotations at that instant to ensure
stability of data. The standard deviations of the continuous annotations are
also provided, averaging about 0.35 across all samples. This averaged standard
deviation is the goal for the first model’s root mean square error (RMSE)
because it means the model’s variation can be accounted for by the human
variation in the dataset. The first 15 seconds of annotations in the data were
dropped due to instability and the 45-second clips of audio are provided with
the dataset.

##  3 Task 1: Predicting Emotion

### Input/Output

The first task is to predict the arousal and valence values of a 0.5-second
clip of audio’s mel spectrogram. The input is a tensor of shape 128 by 44
representing the clips’ mel spectrogram with specific settings (described in
the background). The output will be a two-value tensor representing arousal
and valence.

### Evaluation Metric

The model will use mean squared error (MSE) for its evaluation metric and loss
function. MSE penalizes heavily for deviation as it the square of the output
unit. The target for model loss is 0.09 (or RMSE of 0.3) as described in the
background section. Since human annotations vary by about 0.3 on average on
the same songs, the error produced by the model can be accounted for by human
error in the dataset.

### Hyperparameter Optimization

This model uses the Adam optimizer. Experimenting with hyperparameters, the
learning rate had the highest influence on results, and a low number of hidden
layers performed best. To determine the number of epochs an early stopping
technique was used in which the model automatically ended training after loss
converged and stopped decreasing. A fixed batch size of 58 samples was used,
where each batch contained every clip from one audio file. The best results
found are shown in Table. 1 and Fig. 3. Although decreasing the learning rate
from $5\text{\times}{10}^{-5}$ may slightly improve the convergence of MSE,
the final MSE is very similar whereas training time and epochs increase.

Table 1: Task 1 Best Results Parameter | Value
---|---
Learning Rate | $5\text{\times}{10}^{-5}$
Hidden Size | 20
No. of Modules111Where a ”module” is an nn.LSTM object in the model and layers within a module are stacked | 2
No. of Layers (per module) | 2
No. of Layers (total) | 4
Dropout Probability | 0.1
Epochs | 67
Train MSE | 0.044
Validation MSE | 0.054
Figure 3: Loss graphs (MSE) for training and validation varied with number of
epochs for task 1 using the most optimal hyperparameter found. Shows
convergence for training but slight possible overfitting.

##  4 Task 2: Intelligent Queuing

### Input/Output

The second task is to predict the next pair of arousal and valence values from
an arbitrarily chosen time sequence of length 10. The input is a tensor of
shape 2 by 10 representing the arousal and valence over ten timesteps. The
output will be a two-value tensor representing the next (11th) arousal and
valence point.

### Evaluation Metric

The model will also use MSE for its evaluation metric and loss function for
the same reasons as in Task 1. The target MSE/RMSE are similar due to natural
variation in humans but are expected to be lower in a simpler task.

### Hyperparameter Optimization

Similar to the previous task, hyperparameter optimization was largely
experimental with learning rate and epoch count having a high influence on
results. This model still uses the Adam optimizer and mean squared error,
however, batching is randomized and the task in simpler. The best results
found are shown in Table. 2 and Fig. 4.

Table 2: Task 2 Best Results Parameter | Value
---|---
Learning Rate | $1\text{\times}{10}^{-4}$
Hidden Size | 32
Batch Size | 64
No. of Modules222Where a ”module” is an nn.LSTM object in the model and layers within a module are stacked | 1
No. of Layers (per module) | 2
No. of Layers (total) | 42
Dropout Probability | 0
Epochs | 10
Train MSE | $4\text{\times}{10}^{-4}$
Validation MSE | $5\text{\times}{10}^{-4}$
Figure 4: Loss graphs (MSE) for training and validation varied with number of
epochs for task 2 using the most optimal hyperparameter found. The loss
converges very quickly after the first two epochs.

###  4.1 Linear Regression Approach

Since the LSTM results showed that the task may have been too simple and
revealed a possible exploding gradient issue, instead a linear regression
approach was used. One in which arousal was the dependent variable and the
other in which valence was the dependent variable with both having time as the
independent variable. When the arousal and valence are plotted against time,
they appear linear/constant (Fig. 5), but, zooming in, some clips show
erratic, nonlinear behavior (Fig. 6). The linear regression showed that the
data’s overall trend may be able to be modeled by a regression, however, the
model was not suitable for predicting exact values, especially within the
songs (Fig. 7).

Figure 5: Arousal and valence for one song plotted against time on the z-axis.
Dotted lines represent the centerlines and edges of the circumplex model.
Figure 6: A zoomed view of Figure 5 that shows more erratic and nonlinear
patterns in arousal and valence.  Figure 7: The linear regression results of
one particular song. Shows a good model of the trend but a clear lack of
accuracy around the center of the time axis.

##  5 Results and Conclusions

###  5.1 Task 1

The emotion prediction model got an MSE loss of about 0.055 in validation and
0.044 in training meaning that RMSE are 0.235 and 0.21, respectively. These
RMSE values can be accounted for by the natural variation in even human choice
annotation of this data, indicating the model is doing well, matching human
predictions.

###  5.2 Task 2

The "next value" predictor model showed an MSE of about 0.0004 in training and
0.0005 in validation. These results indicate that the model performed well.
The linear regression model proved to be less effective in predicting exact
values but may be effective in considering general trends.

##  6 Discussion

The ability to automatically determine quantitative values to describe emotion
of a music clip can have uses in the real world. In the medical field, music
is well known as having psychological effects on mood but has also been found
to have effects on other neurological disorders including multiple sclerosis
and Parkinson’s. [9] Commercial applications of this software also include
intelligent queueing of music to maintain a smooth flow of mood in music
playback. A demonstrative implementation of this model is included in the
public repository.333Please reference the GitHub repository for more
information in the demo.py file

##  7 Future Direction

For task 1 a broad, automated hyperparameter search may be able to refine and
optimize hyperparameter for even better regression results. Different batching
strategies and model complexities may also result in an overall superior
model. The second task would benefit from many of the improvements of task 1,
as well as using a statistic to objectively measure the performance of
different models such as the linear regression. Adding variable length time-
sequence input is also a possibility. Implementing this code in an open-source
library that can be used in applications could greatly help those who use it.
The code could also be implemented alongside the queuing algorithms of major
streaming services (e.g. Spotify) to offer improved experiences to users.

##  8 Error Analysis

Limited hyperparameter scope, input format, noise added, and human error could
have resulted in errors in both task one and two’s LSTM models. In model one
it is important to note the slight overfitting occurring indicating that
hyperparameters could likely be further optimized and in model two there may
be an exploding gradient problem. In the second model, the predicted value
tended to hold the arousal and valence constant due to the mostly constant
nature of data, however, in songs with greater changes, the model reflected
those changes. In implementation, adding a tolerance to allow for changes
between songs would avoid holding the arousal and valence constant
unintentionally.

##  9 Data Availability

The Emotion in Music Database (1000 songs) dataset is available online after
filling out a request form.[1, 11]

##  10 Code Availability

The code, models, and results from this dataset have been open-sourced under
the MIT license and are available at https://github.com/etashj/Exploring-and-
Applying-Audio-Based-Sentiment-Analysis.

##  11 Acknowledgments

I would like to thank the developers of the Emotions in Music Database[1, 11]
for making their code open source and free to use as it was of great
importance to the development of this project. I would also like to thank the
developers of the PyTorch[7], NumPy[3], pandas[12], and librosa[5] libraries
for the Python programming language who also made their code open source which
provided the framework for this project.

## References

  * [1] Florian Eyben, Felix Weninger, Florian Gross and Björn Schuller  “Recent Developments in openSMILE, the Munich Open-source Multimedia Feature Extractor”  In _Proceedings of the 21st ACM International Conference on Multimedia_ , MM ’13  Barcelona, Spain: ACM, 2013, pp. 835–838  DOI: 10.1145/2502081.2502224
  * [2] Todor Ganchev, Nikos Fakotakis and Kokkinakis George  “Comparative evaluation of various MFCC implementations on the speaker verification task”  In _Proceedings of the SPECOM_ 1, 2005
  * [3] Charles R. Harris et al.  “Array programming with NumPy”  In _Nature_ 585.7825 Springer ScienceBusiness Media LLC, 2020, pp. 357–362  DOI: 10.1038/s41586-020-2649-2
  * [4] Bochen Li  “Machine Learning for Audio Signals: ECE 272/472 Audio Signal Processing”, PDF document  URL: https://hajim.rochester.edu/ece/sites/zduan/teaching/ece472/lectures/Lecture_MachineLearningForAudio.pdf
  * [5] Brian McFee et al.  “librosa: Audio and Music Signal Analysis in Python”  In _SciPy_ , 2015  URL: https://api.semanticscholar.org/CorpusID:33504
  * [6] Walaa Medhat, Ahmed Hassan and Hoda Korashy  “Sentiment analysis algorithms and applications: A survey”  In _Ain Shams Engineering Journal_ 5.4, 2014, pp. 1093–1113  DOI: https://doi.org/10.1016/j.asej.2014.04.011
  * [7] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Neural Information Processing Systems_ , 2019  URL: https://api.semanticscholar.org/CorpusID:202786778
  * [8] BRADLEY S. PETERSON, JONATHAN POSNER and JAMES A. RUSSELL  “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology”  In _Development and Psychopathology_ 17.3 Cambridge University Press, 2005, pp. 715–734  DOI: DOI: 10.1017/S0954579405050340
  * [9] Alfredo Raglio et al.  “Effects of music and music therapy on mood in neurological patients”  In _World J Psychiatry_ 5.1, 2015, pp. 68–78
  * [10] Maghilnan S and Rajesh Muthu  “Sentiment Analysis on Speaker Specific Speech Data”, 2017  DOI: 10.1109/I2C2.2017.8321795
  * [11] Mohammad Soleymani et al.  “1000 Songs for Emotional Analysis of Music”  In _Proceedings of the 2Nd ACM International Workshop on Crowdsourcing for Multimedia_ , CrowdMM ’13  Barcelona, Spain: ACM, 2013, pp. 1–6  DOI: 10.1145/2506364.2506365
  * [12] The team  “pandas-dev/pandas: Pandas”  Zenodo, 2023  DOI: 10.5281/zenodo.8301632
  * [13] Christine D. Wilson-Mendenhall, Lisa Feldman Barrett and Lawrence W. Barsalou  “Neural Evidence That Human Emotions Share Core Affective Properties” PMID: 23603916  In _Psychological Science_ 24.6, 2013, pp. 947–956  DOI: 10.1177/0956797612464242