# MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient
Electronic Health Records

Xi Sheryl Zhang1, Fengyi Tang2, Hiroko Dodge3,4, Jiayu Zhou2, Fei Wang1
1Department of Healthcare Policy and Research. Weill Cornell Medicine. Cornell
University.
2Department of Computer Science and Engineering. Michigan State University.
3Michigan Alzheimer’s Disease Center, Department of Neurology, University of
Michigan.
4Layton Aging and Alzheimer’s Disease Center, Department of Neurology, Oregon
Health & Science University.
sheryl.zhangxi@gmail.com;tangfeng,jiayuz@msu.edu;hdodge<EMAIL_ADDRESS>
(2019)

###### Abstract.

In recent years, increasingly augmentation of health data, such as patient
Electronic Health Records (EHR), are becoming readily available. This provides
an unprecedented opportunity for knowledge discovery and data mining
algorithms to dig insights from them, which can, later on, be helpful to the
improvement of the quality of care delivery. Predictive modeling of clinical
risk, including in-hospital mortality, hospital readmission, chronic disease
onset, condition exacerbation, etc., from patient EHR, is one of the health
data analytic problems that attract most of the interests. The reason is not
only because the problem is important in clinical settings, but also there are
challenges working with EHR such as sparsity, irregularity, temporality, etc.
Different from applications in other domains such as computer vision and
natural language processing, the labeled data samples in medicine (patients)
are relatively limited, which creates lots of troubles for effective
predictive model learning, especially for complicated models such as deep
learning. In this paper, we propose MetaPred, a meta-learning for clinical
risk prediction from longitudinal patient EHRs. In particular, in order to
predict the target risk where there are limited data samples, we train a meta-
learner from a set of related risk prediction tasks which learns how a good
predictor is learned. The meta-learned can then be directly used in target
risk prediction, and the limited available samples can be used for further
fine-tuning the model performance. The effectiveness of MetaPred is tested on
a real patient EHR repository from Oregon Health & Science University. We are
able to demonstrate that with Convolutional Neural Network (CNN) and Recurrent
Neural Network (RNN) as base predictors, MetaPred can achieve much better
performance for predicting target risk with low resources comparing with the
predictor trained on the limited samples available for this risk.

meta-learning, clinical risk prediction, electronic health records

††journalyear: 2019††copyright: acmcopyright††conference: The 25th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining; August 4–8, 2019;
Anchorage, AK, USA††booktitle: The 25th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining (KDD’19), June 22–24, 2019, Anchorage, AK,
USA††price: 15.00††doi: 10.1145/XXXXXX.XXXXXX††isbn: 978-1-4503-6201-6/19/08

##  1\. Introduction

The recent years have witnessed a surge of interests in healthcare analytics
with longitudinal patient Electronic Health Records (EHR) (Jensen et al.,
2012). Predictive modeling of clinical risk, such as mortality (Sun et al.,
2018; Tang et al., 2018), hospital readmission (Caruana et al., 2015; Shadmi
et al., 2015), onset of chronic disease (Choi et al., 2016), condition
exacerbation (Kerkhof et al., 2015), etc., has been one of the most popular
research topics. This is mainly because 1) accurate clinical risk prediction
models can help the clinical decision makers to identify the potential risk at
its early stage, therefore appropriate actions can be taken in time to provide
the patient with better care; 2) there are many challenges on analyzing
patient EHR, such as sequentiality, sparsity, noisiness, irregularity, etc.
(Wang et al., 2012). Many computational algorithms have been developed to
overcome these challenges, including both conventional approaches (Caruana et
al., 2015) and deep learning models (Choi et al., 2016).

One important characteristic that makes those healthcare problems different
from the applications in other domains, such as computer vision (LeCun et al.,
2015), speech analysis (Deng et al., 2013) and natural language processing
(Young et al., 2018), is that the number of the available sample data set is
typically limited, and typically it is very expensive and sometimes even
impossible for obtaining new samples. For example, for the case of
individualized patient risk prediction, where the goal is to predict a certain
clinical risk for each patient, each data sample corresponds to a patient.
There are in total just 7.5 billion people all over the world, and the number
will be far less if we focus on a specific disease condition. These patients
are also distributed in different continents, different states, different
cities, and different hospitals. The reality is that we only have a small
number of patients available in a specific EHR corpus for training a risk
prediction model. Moreover, the clinical risks we focus on are extraordinarily
complicated. For the majority of the deadly diseases, we are still not clear
about their underlying biological mechanisms and thus the potential treatment
strategies. This means that, in order to learn accurate clinical risk
prediction models, we need to make sufficient use the limited patient samples,
and effectively leverage available knowledge about the clinical risk as well
as predictive models.

Recently, transfer learning (Pan et al., 2010) has been demonstrated as an
effective mechanism to achieve good performance in learning with limited
samples in medical problems. For example, in computer vision, Inception-V3
(Szegedy et al., 2016) is a powerful model for image analysis. Google has
released the model parameters trained on the huge ImageNet data set (Deng et
al., 2009). Esteva et al. (Esteva et al., 2017) adopted such a model as the
starting point, and leveraged a locally collected 130K skin images to fine-
tune the model to discriminate benign vs. malignant skin lesions. They
achieved satisfactory classification performance that is comparable to the
performance of well-trained dermatologists. Similar strategies have also
achieved good performance in other medical problems with different types of
medical images (Gulshan et al., 2016; Kermany et al., 2018). In addition to
computer vision, powerful natural language processing models such as
transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2018) with
parameters trained on general natural language data, have also been fine-tuned
to analyze unstructured medical data (Lee et al., 2019). Because these models
are pre-trained on general data, they can only encode some general knowledge,
which is not specific to medical problems. Moreover, such models are only
available with certain complicated architectures with a huge amount of general
training data. It is difficult to judge how and why such a mechanism will be
effective in which clinical scenarios.

In this paper, we propose MetaPred, a meta-learning framework for low-resource
predictive modeling with patient EHRs. Meta-learning (Thrun and Pratt, 1998;
Ritter et al., 2018) is a recent trend in machine learning aiming at learning
to learn. By low-resource, we mean that only limited EHRs can be used for the
target clinical risk, which is insufficient to train a good predictor by seen
samples of the task themselves. For this scenario, we develop a model agnostic
gradient descent framework to train a meta-learner on a set of prediction
tasks where the target clinical risks are highly relevant. For these tasks, we
choose one of them as the simulated target and the rest as sources. The
parameters of the predictive model will be updated through a step-by-step
sequential optimization process. In each step, an episode of data will be
sampled from the sources and the simulated target to support the updating on
model parameters. To compensate for the optimization-level fast adaptation, an
objective-level adaptation is also proposed. We validate the effectiveness of
MetaPred on a large-scale real-world patient EHR corpus with a set of
cognition related disorders as the clinical risks to be predicted, and
Convolutional Neural Networks (CNN) as well as Long-Short Term Memory (LSTM)
are applied as the predictors because of their popularity in EHR-based
analysis. Additionally, we demonstrate that if we use EHRs in target domains
to fine-tune the learned model, the prediction performance can be further
improved.

The rest of the paper is organized as follows: the problem setup is presented
in Section 2; the proposed framework MetaPred is introduced in Section 3;
experimental results are shown in Section 4 and related works are summarized
in Section 5; finally, conclusion reaches at Section 6.

##  2\. Problem Setup

Figure 1. Illustration of the proposed learning procedure. In this example,
our goal is to predict the risks of Alzheimer’s disease with few labeled
patients, which give rise to a low-resource classification. The idea is to
take advantage of labeled patients from other relevant high-resource domains
and design the learning to transfer workflow with sources and a simulated
target via meta-learning.

In order to introduce our framework, we provide a graphical illustration in
Figure 1. Suppose the target task is the prediction of the onset risk of
Alzheimer’s Disease where we do not have enough training patient samples, and
we want to transfer knowledge from other related disease domains with
sufficient labels such as Mild Cognitive Impairment (MCI) or Dementia.
However, traditional transfer learning would be also constrained by the small
number of training samples, especially for those with complicated neural
networks. Consequently, we take advantage of meta-learning by setting a
simulated target domain for learning to transfer. Though applying meta-
learning settings on the top of low-resource medical records for disease
prediction seems intuitive, how to set up the problem is crucial.

More formally, we consider multiple related disease conditions as the set of
source domains ${\mathcal{S}^{1},\cdots,\mathcal{S}^{K}}$ and a target domain
${\mathcal{T}^{0}}$. This leads to $K+1$ domains in total. In each domain, we
can construct a training data set including the EHRs of both case (positive)
and control (negative) patients. We use the data collection
$\\{(\mathbf{X},\mathbf{y})\\}_{i},i=0,1,\cdots,K$ to denote the features and
labels of the patients in these $K+1$ domains. Our goal is to learn a
predictive model $f$ for the target domain ${\mathcal{T}^{0}}$. In the
following we use $\Theta$ to denote the parameters of $f$. Because only a
limited number of samples are available in $\mathcal{T}^{0}$, we hope to
leverage the data from those source domains, i.e.,
$f=(\mathcal{D}_{\mathcal{S}},\mathbf{X};\Theta)$, where $\mathcal{D}_{S}$
denotes the collection of data samples in the source domains. From the
perspective of domain adaptation (Ben-David et al., 2010), the problem can be
reduced to the design and optimization of model $f$ in an appropriate form of
$\mathcal{D}_{\mathcal{S}}$.

In this section we will mainly introduce how to utilize the source domain data
$\mathcal{D}_{\mathcal{S}}$ in our MetaPred framework. The details on the
design of $f$ will be introduced in the next section. In general, supervised
meta-learning provides models trained by data episodes $\\{\mathcal{D}_{i}\\}$
which is composed of multiple samples. Each $\mathcal{D}_{i}$ is usually split
into two parts according to their labels. We further refer to the domain where
the testing data are from the simulated target domain
$\mathcal{D}_{\mathcal{T}^{s}}$, and it is still one of the source domains.
Followed previous work (Finn et al., 2017a; Ravi and Larochelle, 2016), we
called the training procedure based on this split as meta-train, and the
testing procedure as meta-test.

In summary, the proposed MetaPred framework illustrated in Figure 1 consists
of four steps: (1) constructing episodes by sampling from the source domains
and the simulated target domain; (2) learn the parameters of predictors in an
episode-by-episode manner; (3) fine-tuning the model parameters on the genuine
target domain; (4) predicting the target clinical risk.

Figure 2. The overview of MetaPredworkflow. $\mathcal{D}_{epi}$ is an episode
randomly sampled. $\\{\mathcal{S}^{i}\\}_{i=1}^{K-1}$ denotes source domains
and $\mathcal{T}^{s}$ denotes the simulated target domain. The two gradient
update loops of meta-training process are illustrated. The yellow colored
blocks and arrows are associated with Learner, while the blue ones are
associated with MetaLearner. (“Target loss” is used here instead of “Simulated
Target loss” for simplicity.)

##  3\. The MetaPred Framework

The model-agnostic meta-learning strategy (Finn et al., 2017a) serves as the
backbone of our MetaPred framework. In particular, our goal is to learn a risk
predictor on the target domain. In order to achieve that, we first perform
model agnostic meta-learning on the source domains, where the model parameters
are learned through

(1)
$\displaystyle\Theta^{*}=\mathbf{Learner}({\mathcal{T}^{s}};\mathbf{MetaLearner}({\mathcal{S}^{1},\cdots,\mathcal{S}^{K-1}}))$

where for each data episode, the model parameters are first adjusted through
gradient descents on the objective loss measure on the training data from the
source domains ($\mathbf{MetaLearner}$), and then they will be further
adjusted on the simulated target domain $\mathcal{T}^{s}$
($\mathbf{Learner}$). In the following, we will introduce the learning process
in detail, where the risk prediction model is assumed to be either CNN or
LTSM. First we provide basic neural network prediction models as the options
for $\mathbf{Learner}$. Then we introduce the entire parameter learning
procedure of the proposed MetaPred, including optimization-level adaptation
and objective-level adaptation.

###  3.1. Risk Prediction Models

The EHR can be represented by sequences with multiple visits occurring at
different time points for each patient. At each visit, the records can be
depicted by a binary vector $\mathbf{x}^{t}\in\\{0,1\\}^{\mathcal{|C|}}$,
where $t$ denotes the time point. The values of $1$ indicate the corresponding
medical event occurs at $t$, and 0 otherwise. $\mathcal{C}$ is the vocabulary
of medical events, and $\mathcal{|C|}$ is its cardinality. Thus input of the
predictive models can be denoted as a multivariate time series matrix
$\mathbf{X}_{i}=\\{\mathbf{x}_{i}^{t}\\}_{t=1}^{T_{i}}$, where $i$ is the
patient index and $T_{i}$ is the number of visits for patient $i$. The risk
prediction model is trained to find a transformation mapping from input time
series matrix $\mathbf{X}_{i}$ to the target disease label
$\mathbf{y}_{i}\in\\{0,1\\}^{2}$. This makes the problem a sequence
classification problem.

CNN-based Sequence Learning. There are three basic modules in our CNN based
structure: embedding Layer, convolutional layer and multi-layer perceptron
(MLP). Similar to natural language processing tasks (Kim, 2014), 1-dimensional
convolution operators are used to discover the data patterns along the
temporal dimension $t$. Because the values of medical records at the visits
are distributed in a discrete space, which is sparse and high-dimensional. It
is necessary to place an embedding layer before CNN, to obtain a more compact
continuous space for patient representation. The learnable parameters of the
embedding layer are a weight matrix
$\mathbf{W}_{emb}\in\mathbb{R}^{d\times\mathcal{|C|}}$ and a bias vector
$\mathbf{b}_{emb}\in\mathbb{R}^{d}$, where $d$ is a dimension of the
continuous space. The input vector at each visit $\mathbf{x}_{t}$ is mapped.

The 1-dimensional convolution network employs multiple filter matrices with
one of their dimension fixed as the same as hidden dimension $d$, which can be
denoted as $\mathbf{W}_{conv}\in\mathbb{R}^{l\times d}$. The other filter
dimension $l$ denotes the size of a filter. A max pooling layer is added after
the convolution operation to get the most significant dimensions formed into a
vector representation for each patient. Finally, three MLP layers are used to
produce the risk probabilities as a prediction $\hat{y}_{i}$ for the patient
$i$. To sum, all of the weight matrices, as well as bias vectors in our three
basic modules, make up the whole collection of parameter $\Theta$, which is
optimized through feeding the network patients’ data
$\mathcal{D}=\\{(\mathbf{X}_{i},\mathbf{y}_{i})\\}$.

LSTM-based Sequence Learning. Recurrent Neural Networks are frequently adopted
as a predictive model with great promise in many sequence learning tasks. As
for EHRs, RNN can in principle map from the sequential medical records of
previous inputs that “memory” information that has been processed by the model
previously. A standard LSTM model (Hochreiter and Schmidhuber, 1997) is used
to replace the convolutional layer in the CNN architecture we just introduced.
LSTM weights, which are also parts of $\Theta$, can be summarized into two
mapping matrix as $\mathbf{W}_{h}\in\mathbb{R}^{d\times 4d}$ and
$\mathbf{W}_{x}\in\mathbb{R}^{d\times 4d}$. They are in charge of gates
(input, forget, output) controlling as well as cell state updating. We keep
the same network structures of the embedding layer and MLPs to make CNN and
LSTM comparable for each other.

Learner. With the learned parameter $\Theta$, the prediction probability of an
input matrix $\mathbf{X}_{i}$ is computed by
$\hat{\mathbf{y}}_{i}=f(\mathbf{X}_{i};\Theta)$. The neural networks can be
optimized by minimizing the following objective with a cross-entropy:

(2)
$\displaystyle\mathcal{L}(\Theta)=-\frac{1}{N}\sum_{i=1}^{N}\left(\left(\mathbf{y}_{i}\right)^{T}\log\left(\hat{\mathbf{y}}_{i}\right)+\left(1-\mathbf{y}_{i}\right)^{T}\log\left(1-\hat{\mathbf{y}}_{i}\right)\right)$

where $N$ denotes the patient number in the training data. Similarly, the loss
functions for source and target domains have the same formulation with Eq.
(2), which are denoted as $\mathcal{L}_{\mathcal{S}}$ and
$\mathcal{L}_{\mathcal{T}^{s}}$.

###  3.2. MetaPred Architecture

Optimization-Level Adaptation. In general, meta-learning aims to optimize the
objective over a variety of learning tasks $\mathcal{T}$ which are associated
with the corresponding datasets $\mathcal{D}_{\mathcal{T}}$. The training
episodes $\mathcal{D}_{epi}$ are generated by a data distribution
$p(\mathcal{D}_{\mathcal{T}})$. Then the learning procedure of parameter
$\Theta$ is defined as:

(3)
$\displaystyle\Theta^{*}=\arg\min_{\Theta}\mathbb{E}_{m}\mathbb{E}_{\mathcal{D}_{epi}^{m}\sim
p(\mathcal{D}_{\mathcal{T}})}\mathcal{L}_{\Theta}(\mathcal{D}_{\mathcal{T}})$

where $m$ episodes of training samples are used in the optimization.
$\mathcal{L}_{\Theta}$ is the loss function that might take different
formulations depending on the different strategies to design a meta-learner.
As it is claimed in meta-learning, the models should be capable of tackling
the unseen tasks during testing stages. In order to achieve this goal, the
loss function for one episode can be further defined as the following form:

(4)
$\displaystyle\mathcal{L}_{\Theta}=\frac{1}{|\mathcal{D}_{epi}^{te}|}\sum_{(\mathbf{X}_{i},\mathbf{y}_{i})\in\mathcal{D}_{epi}^{te}}\mathcal{L}_{\Theta}\left((\mathbf{X}_{i},\mathbf{y}_{i});\mathcal{D}_{epi}^{tr}\right)$

where $\mathcal{D}_{epi}^{tr}$ and $\mathcal{D}_{epi}^{te}$ are the two parts
of a sample set that simulated training and testing in each episode as we
introduced previously. It is worth to note that Eq. (4) is a loss decided by
the prediction qualities of samples in $\mathcal{D}_{epi}^{te}$. The model-
agnostic meta-learning (MAML) (Finn et al., 2017a) provides us a parameter
initialization scheme for $\Theta$ in Eq. (4) by taking full advantage of
$\mathcal{D}_{epi}^{tr}$. It assumes that there should be some internal
representations are more transferable than others, which could be discovered
by an inner learning procedure using $\mathcal{D}_{epi}^{tr}$. Based on the
essential idea, we show the underlying mechanism of model-agnostic meta-
learning fits the problem of transferring knowledge from source domains to a
low-resource target domain very well, which can be used in solving the risk
prediction problem of several underdiagnosed diseases.

Figure 2 illustrates the architecture of the proposed MetaPred. The general
meta-learning algorithms generate episodes over task distributions and shuffle
the tasks to make each task could be a part of $\mathcal{D}_{epi}^{tr}$ or
$\mathcal{D}_{epi}^{te}$. Instead, we define the two disjoint parts of the
episode as source domains and a target domain to satisfy a transfer learning
setting. To construct a single episode $\mathcal{D}_{epi}$ in the meta-
training process, we sample data via
$\\{(\mathbf{X}_{\mathcal{S}^{i}},\mathbf{y}_{\mathcal{S}^{i}})\\}\sim
p(\mathcal{D}_{\mathcal{S}^{i}})$ and
$\\{(\mathbf{X}_{\mathcal{T}^{s}},\mathbf{y}_{\mathcal{T}^{s}})\\}\sim
p(\mathcal{D}_{\mathcal{T}^{s}})$ respectively. In order to optimize $\Theta$
that can quickly adapt to the held-out samples in target domain, the inner
learning procedure should be pushed forward by the supervise information of
the source samples. To meet this requirement, the following gradient update
occurs:

(5)
$\displaystyle\Theta^{\prime}=\Theta-\alpha\nabla_{\Theta}\sum_{i}^{K-1}\mathcal{L}_{\mathcal{S}^{i}}$

where $\mathcal{L}_{\mathcal{S}^{i}},i=1,\cdots,K-1$ are loss functions of
source domains. $\alpha$ is a hyperparameter controlling the update rate. The
source loss is computed by
$\mathcal{L}_{\mathcal{S}^{i}}=\mathcal{L}(\mathbf{y}_{\mathcal{S}^{i}},f(\mathbf{X}_{\mathcal{S}^{i}},\Theta)$.
From Eq. (5) we can observe that it is a standard form gradient descent
optimization. In practice, we will repeat this process $k$ times, then output
the $\Theta^{\prime}$ as an initial parameter for the simulated target domain.
The inner learning can be view as an Inner-Loop which is shown in Figure 2.

Once we set $\Theta=\Theta^{\prime}$ before the update step of the simulated
target domain, the minimize problem defined by the loss given in Eq. (4)
becomes:

(6)
$\displaystyle\min_{\Theta}\mathcal{L}_{\mathcal{T}^{s}}(f_{\Theta^{\prime}})=\min_{\Theta}\sum_{\mathcal{D}_{epi}^{\mathcal{T}^{s}}\sim
p(\mathcal{D}_{\mathcal{T}^{s}})}\mathcal{L}\left(\mathbf{y}_{\mathcal{T}^{s}},f(\mathbf{X}_{\mathcal{T}^{s}},\Theta^{\prime})\right)$

where
$\mathcal{D}_{\mathcal{T}^{s}}=\\{(\mathbf{X}_{\mathcal{T}^{s}},\mathbf{y}_{\mathcal{T}^{s}})\\}$.
Given the loss form of $\mathcal{L}_{\mathcal{T}^{s}}$ in the simulated target
domain, it is computed by the output parameter $\Theta^{\prime}$ obtain via
inner gradient update in Eq. (5). Then, the meta-optimization using
$\mathcal{D}_{\mathcal{T}^{s}}$ is performed with:

(7)
$\displaystyle\Theta=\Theta-\beta\nabla_{\Theta}\mathcal{L}_{\mathcal{T}^{s}(f_{\Theta^{\prime}})}$

where $\beta$ is the meta-learning rate. Hence, the simulated target loss
involves an Outer-Loop for gradient updating. Compared to the standard
gradient updating in Eq. (5), the gradient-like term in Eq. (7) essentially
resorts to a gradient through a gradient that can be named as meta-gradient.
Accordingly, the entire learning procedure can be viewed as: iteratively
transfer the parameter $\Theta$ learned from source domains through utilizing
it as the initialization of the parameter that needs to be updated in the
simulated target domain.

To build end-to-end risk prediction models with the model-agnostic gradient
updating, we use the deep neural network structures that are trained using
medical records $\mathbf{X}$ and diagnosis results $\mathbf{y}$ described in
Section 3.1. The objectives for both source and simulated target are set as
cross-entropy introduced in Eq. (2). One interesting point is that all the
parameters of source domains and simulated target domains are tied, with
different stages to update. The colors in Figure 2 provides an indication
about the aforementioned two kinds of gradient pass.

Algorithm 1 MetaPred Training

0: Source domains $\mathcal{S}^{i}$; Simulated target domain
$\mathcal{T}^{s}$;

0: Hyperparameters $\alpha,\beta,\mu$;

1: Initialize model parameter $\Theta$ randomly

2: while Outer-Loop not done do

3: Sample batch of episodes $\\{\mathcal{D}_{epi}\\}$ from
$\mathcal{D}_{\mathcal{S}^{i}}$ and $\mathcal{D}_{\mathcal{T}^{s}}$

4: while Inner-Loop not done do

5:
$\\{(\mathbf{X}_{\mathcal{S}^{i}},\mathbf{y}_{\mathcal{S}^{i}})\\}_{i=1}^{K-1},\\{(\mathbf{X}_{\mathcal{T}^{s}},\mathbf{y}_{\mathcal{T}^{s}})\\}=\\{\mathcal{D}_{epi}\\}$

6: Compute
$\mathcal{L}_{\mathcal{S}^{i}}=\mathcal{L}(\mathbf{y}_{\mathcal{S}^{i}},f(\mathbf{X}_{\mathcal{S}^{i}},\Theta)),i=1,\cdots,K-1$

7: Parameter fast adaption with gradient descent:

8:
$\Theta^{\prime}=\Theta-\alpha\nabla_{\Theta}\sum_{i}^{K-1}\mathcal{L}_{\mathcal{S}^{i}}$

9: end for

10: Compute
$\mathcal{L}_{\mathcal{T}^{s}}=\mathcal{L}(\mathbf{y}_{\mathcal{T}^{s}},f(\mathbf{X}_{\mathcal{T}^{s}},\Theta^{\prime}))$

11: Update
$\Theta=\Theta-\beta\nabla_{\Theta}(\mathcal{L}_{\mathcal{T}^{s}}+\mu\sum_{i}^{K-1}\mathcal{L}_{\mathcal{S}^{i}})$
using Adam

12: end while

Objective-Level Adaptation. While MAML provides an effective transferable
parameter learning scheme for disease risk prediction in the low-resource
situation, it cannot ensure sufficiently transferring the critical knowledge
from the source domain. On the one hand, meta-learning generally encourages
that the simulated target task could be randomly generated, and their model
could be adapted to a large or infinite number of tasks (Finn et al., 2017a;
Santoro et al., 2016; Vinyals et al., 2016). Different from these works,
transfer learning often requires to capture domain shifts. To do so, the
simulated target that is used in learning to transfer cannot be randomly
sampled.

On the other hand, the task distribution is a common decisive factor of the
success for meta-learning. In other words, the distributions of the
investigated source and target domains should not be too diverse. In real-
world healthcare scenario, however, patients who suffering difference diseases
might have medical records at various visits with heterogeneity. In this case,
it is difficult to meta-learn during optimization loops. To alleviate this
problem, we propose to enhance some guarantee from the objective-level in
predictive modeling so that the scarcity of the fast adaptation in the
optimization-level can be compensated. In particular, we propose to improve
the objective by incorporating supervision from source domains. The final
objective of MetaPred is given in the mathematical form as:

(8)
$\displaystyle\mathcal{L}_{\mathcal{T}}(f_{\Theta^{\prime}})=\mathcal{L}_{\mathcal{T}^{s}}(f_{\Theta^{\prime}})+\mu\sum_{i}^{K-1}\mathcal{L}_{\mathcal{S}^{i}}(f_{\Theta})$
$\displaystyle=\sum_{\mathcal{D}_{epi}^{\mathcal{T}^{s}}}\mathcal{L}\left(\mathbf{y}_{\mathcal{T}^{s}},f(\mathbf{X}_{\mathcal{T}^{s}},\Theta^{\prime})\right)+\mu\sum_{i}^{K-1}\sum_{\mathcal{D}_{epi}^{\mathcal{S}^{i}}}\mathcal{L}\left(\mathbf{y}_{\mathcal{S}^{i}},f(\mathbf{X}_{\mathcal{S}^{i}},\Theta)\right)$

where
$\\{(\mathbf{X}_{\mathcal{S}^{i}},\mathbf{y}_{\mathcal{S}^{i}})\\}_{i=1}^{K-1}$
is a collection of medical records matrix and label vectors of source domains.
$\mathcal{D}_{epi}^{T^{s}}$ and $\mathcal{D}_{epi}^{S^{i}}$ are samples from
the source domain and the simulated target domain in episode
$\mathcal{D}_{epi}$, respectively. Hyperparameter $\mu$ balances the
contributions of the sources and simulated target in the meta-learn process.
Note that the parameter of source loss is $\Theta$ but not $\Theta^{\prime}$,
as there is no need to conduct fast adaptation for source domain. Now the
newly designed meta-gradient is updated by the following equation:

(9)
$\displaystyle\Theta=\Theta-\beta\nabla_{\Theta}(\mathcal{L}_{\mathcal{T}^{s}}+\mu\sum_{i}^{K-1}\mathcal{L}_{\mathcal{S}^{i}})$

So far the main architecture of MetaPred is introduced. With the incorporated
source loss on the basis of general meta-learning, our parameter learning
process need to be redefined as:

(10)
$\displaystyle\Theta^{*}=\mathbf{Learner}\left({\mathcal{T}^{s}},\\{\mathcal{S}^{i}\\}_{i}^{K-1};\mathbf{MetaLearner}(\\{\mathcal{S}^{i}\\}_{i}^{K-1})\right)$

The Algorithm 1 and Algorithm 2 are outlines of meta-training and meta-testing
of the MetaPred framework. Similar to meta-training, episodes of the test set
are consist of samples from the source domain and genuine target domain. The
procedure in meta-test shows how to get a risk prediction for the given low-
resource disease by a few gradient steps. The test set of the target disease
domain is used to construct the meta-test episodes for the model evaluation.
Since MetaPred is model-agnostic, the gradient updating scheme can be easily
extended to more sophisticated neural networks including various attention
mechanisms or gates with prior medical knowledge (Choi et al., 2017; Baytas et
al., 2017).

Algorithm 2 MetaPred Testing

0: Source domains $\mathcal{S}^{i}$; target domain $\mathcal{T}^{0}$;

0: Learned parameter $\Theta$;

1: Sample from $\mathcal{D}_{\mathcal{S}^{i}}$ to construct testing episodes
$\\{\mathcal{D}_{epi}\\}$

2:
$\\{(\mathbf{X}_{\mathcal{S}^{i}},\mathbf{y}_{\mathcal{S}^{i}})\\}_{i=1}^{K-1},\\{(\mathbf{X}_{\mathcal{T}^{0}},\mathbf{y}_{\mathcal{T}^{0}})\\}=\\{\mathcal{D}_{epi}\\}$

3: Compute
$\mathcal{L}_{\mathcal{S}^{i}}=\mathcal{L}(\mathbf{y}_{\mathcal{S}^{i}},f(\mathbf{X}_{\mathcal{S}^{i}},\Theta)),i=1,\cdots,K-1$

4: Parameter fast adaption with gradient descent:

5:
$\Theta^{\prime}=\Theta-\alpha\nabla_{\Theta}\sum_{i}^{K-1}\mathcal{L}_{\mathcal{S}^{i}}$

6: Evaluate predicted results of
Learner$(\\{(\mathbf{X}_{\mathcal{T}^{0}},\mathbf{y}_{\mathcal{T}^{0}})\\};\Theta^{\prime})$

##  4\. Experiments

Table 1. Statistics of datasets with disease domains. Domain | Case | Control | # of visit | Ave. # of visit
---|---|---|---|---
MCI | 1,965 | 4,388 | 161,773 | 22.24
Alzheimer’s | 1,165 | 4,628 | 136,197 | 20.73
Parkinson’s | 1,348 | 3,588 | 105,053 | 20.01
Dementia | 3,438 | 1,591 | 98,187 | 18.06
Amnesia | 2,974 | 4,215 | 180,091 | 21.60
Table 2. Performance on the disease classification tasks. The simulated target domain for three mainly investigated diseases are set as $\textbf{Alzheimer}\sim\textbf{MCI}$, $\textbf{MCI}\sim\textbf{Alzheimer}$, and $\textbf{MCI}\sim\textbf{Parkinson}$ (A is a simulated target and B is a target if $\textbf{A}\sim\textbf{B}$). Training Data | Model | MCI | Alzheimer’s Disease | Parkinson’s Disease
---|---|---|---|---
AUCROC | F1 Score | AUCROC | F1 Score | AUCROC | F1 Score
Fully Supervised | LR | 0.5861 (.01) | 0.3813 (.02) | 0.5369 (.01) | 0.2216 (.02) | 0.7504 (.01) | 0.6391 (.02)
$k$NN | 0.6106 (.01) | 0.4540 (.01) | 0.6713 (.02) | 0.4686 (.03) | 0.7599 (.01) | 0.6403 (.01)
RF | 0.6564 (.01) | 0.4998 (.01) | 0.6300 (.02) | 0.4111 (.04) | 0.7750 (.01) | 0.6898 (.02)
MLP | 0.6515 (.01) | 0.5077 (.01) | 0.6639 (.02) | 0.4901 (.03) | 0.7958 (.02) | 0.7027 (.01)
CNN | 0.6999 (.01) | 0.5816 (.02) | 0.6755 (.03) | 0.4935 (.04) | 0.7980 (.01) | 0.7265 (.02)
LSTM | 0.6874 (.01) | 0.5666 (.02) | 0.6902 (.01) | 0.5316 (.02) | 0.8041 (.02) | 0.7241 (.02)
Low-Resource | Meta-CNN | 0.7624 (.02) | 0.6992 (.02) | 0.7682 (.01) | 0.6434 (.03) | 0.7604 (.02) | 0.6737 (.03)
Meta-LSTM | 0.7876 (.02) | 0.7225 (.02) | 0.7464 (.02) | 0.6170 (.03) | 0.7532 (.02) | 0.6753 (.03)
Fully Fine-Tuned | Meta-CNN | 0.8470 (.01) | 0.7888 (.02) | 0.8461 (.01) | 0.7375 (.01) | 0.8343 (.01) | 0.7406 (.01)
Meta-LSTM | 0.8477 (.01) | 0.7963 (.02) | 0.8232 (.01) | 0.7364 (.01) | 0.8172 (.01) | 0.7291 (.02)

###  4.1. Dataset

In this section, experimental results on a real-world EHR dataset are
reported. The data warehouse we used in experiments is the research data
warehouse (RDW) from Oregon Health & Science University (OHSU) Hospital. The
data warehouse which contains the EHR of over 2.5 million patients with more
than 20 million patient encounters, is mined by Oregon Clinical and
Translational Research Center (OCTRI). For certain conditions, we may not have
sufficient patients for training and testing. In our study, we selected the
conditions including more than $1,000$ cases (MCI, Alzheimer’s disease,
Parkinson’s disease, Dementia, and Amnesia) as the different tasks in the
multi-domain setting. For each domain, controls are patients suffering other
cognitive disorders, which makes the classification tasks difficult and
meaningful in practice. Also, Dementia and Amnesia are used as source domains,
while the more challenging tasks MCI, Alzheimer, Parkinson are set as target
domains.

We matched the case and controls by requiring their age difference within a
5-year range so that the age distributions between the case group and control
group are consistent. For each patient, we set a 2-year observation window to
collect the training data, and the prediction window is set to half a year
(i.e., we are predicting after half a year the onset risk of those
conditions). In our experiments, only patient diagnoses histories are used,
which include 10,989 distinct ICD-9 codes in total. We further mapped them to
their first three digits, which ends up with 1,016 ICD-9 group codes. The data
statistics are summarized in Table 1.

###  4.2. Experimental Setup

Metric & Models for comparison. In our experiments, we take the AUROC (area
under receiver operating characteristic curve) and F1 Score as the prediction
performance measures. We compare the performance of the MetaPred framework
with the following approaches established on the target task.

(a) MCI

(b) Alzheimer’s Disease

(c) Parkinson’s Disease

Figure 3. Results with respect to different levels of labeled data resource
used in fine-tuning for target domains.

_Supervised classification models._ Three traditional classification models
without considering any sequential EHR information, including Logistic
Regression (LR), $k$-Nearest Neighbors algorithm ($k$-NN), and Random Forest
(RF), are implemented as baselines, where the patient vectors are formed by
counting the frequencies of specific diagnosis codes during the observation
window. Deep learning models, including Embedding Layer-MLP and Embedding
Layer-CNN/LSTM-MLP architectures are implemented as baselines.

_Fine-tuned models._ For the adaptation to a target domain, training data of
target domains can be used in fine-tuning an established meta-learning model
based on sources. Among the basic blocks of the built networks, we consider
fine-tuning MLP layers meanwhile freeze the embedding layer and CNN/LSTM
blocks. Therefore, MLP can be viewed as a task-specific architecture leaned
based on the corresponding target.

_Low-Resources models._ Since there are no prior efforts focusing on the
critical problem of low-resource medical records. We propose two variants of
MetaPred to verify its feasibility and effectiveness. Depends on the choice of
modules for sequencing learning, we build Meta-CNN and Meta-LSTM to predict
disease risks with limited target samples. Specifically, patients in the true
target domain are not used in generating the episodes during meta-training,
which makes our setup satisfying the meta-learning tasks. Then a small part of
the training target set is employed to fine-tune the learned models. We keep
this ratio as $20\%$ to simulate low-resource situations.

To show the superior of the parameter transferable ability, we compare the
performance of MetaPred with a basic parameter transfer learning algorithm
(Pan et al., 2010; Lawrence and Platt, 2004), which solves the following
posterior maximization problem:

(11)
$\displaystyle\arg\max_{\Theta}\Sigma_{(\mathbf{X},y)\in\mathcal{D}_{\mathcal{T}}}\log
p(y|\mathbf{X},\Theta)-\gamma\|\Theta-\Theta^{0}\|$

where $\Theta^{0}$ is an initial parameter setting for the target domain. The
norm term gives a prior distribution of parameters and constraints that the
learned model for target task should not deviate too much from the one learned
from source tasks. The transfer learning models are named TransLearn. In
addition, multitask learning methods (Collobert and Weston, 2008; Caruana,
1997) are employed to be another comparison in the limited-resource scenario.
In particular, we fix the bottom layers and use domain-specific MLP in the
multitask baseline MultiLearn. For a fair comparison, the above approaches are
all evaluated by held-out test sets of the target domains.

Implementation Details and Model Selection. For all above algorithms, 20%
patients of the labeled patients are used as a test set for the three main
tasks and train models on the remaining 80%. We randomly split patients with
this ratio for target domain and run experiments five times. The average
performance is reported. The deep learning approaches including the proposed
MetaPred are implemented with Tensorflow. The network architectures of CNN and
LSTM, as well as other hyperparameters are tuned by the $5$-fold cross-
validation. In detail, The hidden dimensions of embedding layer and $2$ fully
connected layers are set as $d_{emb}=256$ and $d_{mlp}=128$. The vocabulary
size is consistent with ICD-9 diagnosis codes, which is grouped as
$d_{vol}=1017$ including $1$ padding index. The sequence length is chosen
according to the average number of visit per patient in Table 1. Batch
normalization (Ioffe and Szegedy, 2015) and layer normalization (Ba et al.,
2016) are employed based on CNN and LSTM respectively. We keep the same
network configurations for single task models and meta-learning models. We use
Adam (Kingma and Ba, 2014) optimizer with a batch size of $32$ episodes to
compute the meta-gradient. In each episode, the number of patients used for
each domain is set at $8$. The proposed MetaPred is trained on machines with
NVIDIA TESLA V100 GPUs. The source code of MetaPred is publicly available at
https://github.com/sheryl-ai/MetaPred.

###  4.3. Performance Evaluation

Performance on Clinical Risk Prediction. The performance of compared
approaches on three mainly investigated risk prediction tasks are presented in
Table 2. According to how many training data used in the target domain, there
are full supervised baselines including traditional classifiers and deep
predictive models, our proposed methods Meta-CNN/LSTM partially using the
training data in fine-tuning, as well as the fully fine-tuned MetaPred models.
The medical knowledge about cognitive diseases suggests us that MCI and
Alzheimer are fairly difficult to be distinguished with other relevant
disorders. Nevertheless, the symptoms of Parkinson’s Disease sometimes are
obvious to be recognized, which makes it a relatively easier task.

From Table 2 we can observe that results obtained by LR, $k$NN, RF, and neural
networks cannot achieve a satisfying classification performance through merely
modeling the target tasks of MCI and Alzheimer. Our method Meta-CNN/LSTM
perform better than single task models even with only $20\%$ labeled target
samples in fine-tuning. The AUC of MetaPred reaches at $0.7876\pm.02$ and
$0.7682\pm.01$ while their corresponding single-task versions only have
$0.6874\pm.01$ and $0.6755\pm.03$. As for Parkinson, because of the
insufficient labeled data, the results of low-resource cannot beat CNN/LSTM.
It also indicates that the domain shift exists in real-world disease
predictions. Under the fully fine-tuned setting, the labels of targets are the
same as the fully supervised setting. MetaPred achieves significant
improvements on all the three classification tasks in terms of AUC and F1
Score.

Comparisons at the different resource levels. In order to show the superiority
of MetaPred in the transferability with multiple domains, transfer learning
and multitask learning methods are used in comparisons. Figure 3 shows F1
Score results giving labeled targets samples at the percentage
{$20\%,40\%,60\%,80\%,100\%$} of the available training data in target domain.
For the transfer learning model TransLearn in Eq.(11), we tried various tasks
as source domains and finally used the setting
$\textbf{Alzheimer}\sim\textbf{MCI}$, $\textbf{MCI}\sim\textbf{Alzheimer}$,
and $\textbf{MCI}\sim\textbf{Parkinson}$ where the best performance achieved.
Meanwhile, MultiLearn models are compared with the same level of supervision
in the three given target tasks. We randomly picked the labeled data from
training set five times, and the mean and variance are presented in Figure 3.
We adopt CNN as the predictive model for the compared methods here. As we can
see, MetaPred outperforms TransLearn and MultiLearn on all of the tasks. The
gap is large for MCI and Alzheimer especially when the labeled data are low.
The TransLearn method can also perform well on the Parkinson task due to their
homogeneity in several symptoms. Overall, the fast adaptation in both
optimization-level and objective-level leads to more robust prediction results
under low-resource circumstances.

(a) Meta-CNN vs. MAML-CNN

(b) Meta-LSTM vs. MAML-LSTM

Figure 4. Comparison between MetaPred and MAML in terms of performance curve
along with the learning procedures (Results on Alzheimer’s Disease).

(a) MCI

(b) Alzheimer’s Disease

(c) Parkinson’s Disease

Figure 5. Results with respect to different combinations of source disease
domains. The best results among different source domains are reported for the
transfer learning method (Compared methods are all under the low-resource
setting). Figure 6. Visualization using a t-SNE plot of patient representation
in a $2$ dimensional space. Node denotes patient suffering cognition disorders
we studied. Color indicates the associated domains.

MetaPred vs. MAML. To demonstrate the effectiveness of the proposed MetaPred
learning procedure and to empirically certify the rationality of objective-
level adaptation, we compare it with the state-of-the-art meta-learning
algorithm MAML (Finn et al., 2017a). Experimental results of this comparison
are shown in Figure 4. To simulate the low-resource scenario, both MetaPred
and MAML use all the available samples from sources and a simulated target for
meta-train and $20\%$ labeled target patients in fine-tuning. To make the
comparison fair, we use the same sets of labeled patients in the evaluation.
The experiments are repeated five times, and the averaged performance with the
confidence interval set as $95\%$ are given. Figure 4 gives results in terms
of AUC and F1 Score for Alzheimer’s Disease classification using both CNN and
LSTM as the base predictive models. Along with the training iterations, the
metric scores of both MetaPred and MAML converge to a stable value, suggesting
the stability of the meta-optimization. Our method MetaPred achieve better
performance in the disease risk prediction tasks by incorporating the
supervised information of source domain.

Impact of Source Domain. In Figure 5, we vary the source domains as {DM, PD,
AM}, {DM, PD, AM}, and {AD, DM, AM}111AD, PD, DM, AM are abbreviations of
Alzheimer’s Disease, Parkinson’s Disease, Dementia, and Amnesia, respectively.
and show the F1 Score results for MCI, Alzheimer, and Parkinson, respectively.
TransLearn is used as a baseline here. Similarly, the simulated targets are
set as $\textbf{Alzheimer}\sim\textbf{MCI}$,
$\textbf{MCI}\sim\textbf{Alzheimer}$, and
$\textbf{MCI}\sim\textbf{Parkinson}$. Once the simulated target is fixed, we
first evaluate the source domain one-by-one, then feed all of them through
episode generator in meta-train. Compared to TransLearn, the variants of
MetaPred generally performs better on the basis of both CNN and LSTM.
Intuitively, using samples from more source domains leads to a more
comprehensive representation space and thus a better prediction result on
targets, which is verified by Figure 5 very well. Besides, source domains have
an influence on the performance largely, especially for MCI and Alzheimer. For
example, the largest gap of F1 Score could be close to $0.25$ in MCI
prediction. The analysis helps us to choose the source domain according to
their performance on the target predictions. That is, Amnesia always benefits
more as a source domain whereas Parkinson benefits less compared to other
sources.

Visualization. Figure 6 provides the visualization results. The
representations learned before the last MLP layer of MetaPred can be extracted
as high-level features for patients. The feature dimension is $128$ as we
aforementioned. During the representation learning, we hold-out $518$ cases
from each domain, and build a MetaPred upon the rest of the data. Then, the
held-out patients are clustered via t-SNE based on the outputted
representations. It is shown that the five diseases are separated quite well
and suggests that MetaPred generates meaningful representations for patients
in several relevant domains.

##  5\. Related Work

Meta-learning, also known as learning to learn (Andrychowicz et al., 2016;
Thrun and Pratt, 1998; Lake et al., 2015), aims to solve a learning problem in
the target task by leveraging the learning experience from a set of related
tasks. Meta-learning algorithms deal with the problem of efficient learning so
that they can learn new concepts or skills fast with just a few seen examples.
Meta-learning algorithms have been recently explored on a series of topics
including few-shot learning (Ravi and Larochelle, 2016; Vinyals et al., 2016;
Santoro et al., 2016), reinforcement learning (Finn et al., 2017a; Ritter et
al., 2018) and imitation learning (Finn et al., 2017b). One scheme of meta-
learning is to incorporate learning structures of data points by distance
functions (Koch et al., 2015) or embedding networks (Vinyals et al., 2016;
Snell et al., 2017) such that the classifier can adapt to accommodate unseen
tasks in training. Another scheme is basically optimization-based which is
training a gradient procedure and applied it on a learner directly (Ravi and
Larochelle, 2016; Andrychowicz et al., 2016; Finn et al., 2017a; Santoro et
al., 2016). Both of the schemes could be summarized as the design and
optimization of a function $f$ which gives predictions for the unseen testing
data $\mathbf{X}_{test}$ with training episodes $\mathcal{D}_{epi}$ and
parameter collection $\Theta$. Specifically, model-agnostic meta-learning
(Finn et al., 2017a) aims to learn a good parameter initialization for the
fast adaptation of testing tasks. It has gained successes in applications such
as robotic (Clavera et al., 2018; Finn et al., 2017b) and neural machine
translation (Gu et al., 2018).

However, the application of meta-learning in healthcare has rarely been
explored, despite the fact that most of the medical problems are resource-
limited. Consequently, we propose MetaPred to address the general problem of
clinical risk predictions with low-resource EHRs.

##  6\. Conclusion

In this paper, we propose an effective framework MetaPred that can solve the
low-resource medical records problem in clinical risk prediction. MetaPred
leverages deep predictive modeling with the model agnostic meta-learning to
exploit the labeled medical records from high-resource domain. For the purpose
of designing a more transferable learning procedure, we introduce a objective-
level adaptation for MetaPred which not only take advantage of fast adaptation
from optimization-level but also take the supervision of the high-resources
domain into account. Extensive evaluation involving $5$ cognitive diseases is
conducted on real-world EHR data for risk prediction tasks under various
source/target combinations. Our results demonstrated the superior performance
of MetaPred with limited patient EHRs, which can even beat fully supervised
deep neural networks for the challenging risk prediction tasks of MCI and
Alzheimer. For future clinical study, comprehensive longitudinal records more
than $5$ years will be explored for cognition related disorders.

## Acknowledgement

The research is supported by NSF IIS-1750326, ONR N00014-18-1-2585, Oregon
Alzheimer’s Disease Center and Michigan AD center grants NIH P30AG008017 and
NIH P30AG053760.

## References

  * (1)
  * Andrychowicz et al. (2016) Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. 2016.  Learning to learn by gradient descent by gradient descent. In _NIPS_.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016\.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ (2016).
  * Baytas et al. (2017) Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. 2017\.  Patient subtyping via time-aware LSTM networks. In _KDD_.
  * Ben-David et al. (2010) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010.  A theory of learning from different domains.  _Machine learning_ 79, 1-2 (2010).
  * Caruana (1997) Rich Caruana. 1997\.  Multitask learning.  _Machine learning_ 28, 1 (1997).
  * Caruana et al. (2015) Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015\.  Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In _KDD_.
  * Choi et al. (2017) Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. 2017.  GRAM: graph-based attention model for healthcare representation learning. In _KDD_. 787–795.
  * Choi et al. (2016) Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. 2016.  Using recurrent neural network models for early detection of heart failure onset.  _JAMIA_ 24, 2 (2016).
  * Clavera et al. (2018) Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. 2018.  Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning.  (2018).
  * Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008.  A unified architecture for natural language processing: Deep neural networks with multitask learning. In _ICML_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _CVPR_.
  * Deng et al. (2013) Li Deng, Jinyu Li, Jui-Ting Huang, Kaisheng Yao, Dong Yu, Frank Seide, Michael L Seltzer, Geoffrey Zweig, Xiaodong He, Jason D Williams, et al. 2013\.  Recent advances in deep learning for speech research at Microsoft.. In _ICASSP_ , Vol. 26.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Esteva et al. (2017) Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017.  Dermatologist-level classification of skin cancer with deep neural networks.  _Nature_ 542, 7639 (2017).
  * Finn et al. (2017a) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017a.  Model-agnostic meta-learning for fast adaptation of deep networks. In _ICML_.
  * Finn et al. (2017b) Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. 2017b.  One-Shot Visual Imitation Learning via Meta-Learning. In _Conference on Robot Learning_. 357–368.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018.  Meta-Learning for Low-Resource Neural Machine Translation. In _EMNLP_.
  * Gulshan et al. (2016) Varun Gulshan, Lily Peng, Marc Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. 2016\.  Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.  _JAMA_ 316, 22 (2016).
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997).
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In _ICML_.
  * Jensen et al. (2012) Peter B Jensen, Lars J Jensen, and Søren Brunak. 2012\.  Mining electronic health records: towards better research applications and clinical care.  _Nature Reviews Genetics_ 13, 6 (2012).
  * Kerkhof et al. (2015) Marjan Kerkhof, Daryl Freeman, Rupert Jones, Alison Chisholm, and David B Price. 2015.  Predicting frequent COPD exacerbations using primary care data.  _International journal of chronic obstructive pulmonary disease_ 10 (2015).
  * Kermany et al. (2018) Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. 2018\.  Identifying medical diagnoses and treatable diseases by image-based deep learning.  _Cell_ 172, 5 (2018).
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_ (2014).
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Koch et al. (2015) Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015\.  Siamese neural networks for one-shot image recognition. In _ICML deep learning workshop_ , Vol. 2.
  * Lake et al. (2015) Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015\.  Human-level concept learning through probabilistic program induction.  _Science_ 350, 6266 (2015).
  * Lawrence and Platt (2004) Neil D Lawrence and John C Platt. 2004.  Learning to learn with the informative vector machine. In _ICML_.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep learning.  _nature_ 521, 7553 (2015).
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: pre-trained biomedical language representation model for biomedical text mining.  _arXiv preprint arXiv:1901.08746_ (2019).
  * Pan et al. (2010) Sinno Jialin Pan, Qiang Yang, et al. 2010\.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ 22, 10 (2010), 1345–1359.
  * Ravi and Larochelle (2016) Sachin Ravi and Hugo Larochelle. 2016.  Optimization as a model for few-shot learning.  (2016).
  * Ritter et al. (2018) Samuel Ritter, Jane Wang, Zeb Kurth-Nelson, Siddhant Jayakumar, Charles Blundell, Razvan Pascanu, and Matthew Botvinick. 2018.  Been There, Done That: Meta-Learning with Episodic Recall. In _ICML_.
  * Santoro et al. (2016) Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016\.  Meta-learning with memory-augmented neural networks. In _ICML_.
  * Shadmi et al. (2015) Efrat Shadmi, Natalie Flaks-Manov, Moshe Hoshen, Orit Goldman, Haim Bitterman, and Ran D Balicer. 2015\.  Predicting 30-day readmissions with preadmission electronic health record data.  _Medical care_ 53, 3 (2015).
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017\.  Prototypical networks for few-shot learning. In _NIPS_.
  * Sun et al. (2018) Mengying Sun, Fengyi Tang, Jinfeng Yi, Fei Wang, and Jiayu Zhou. 2018.  Identify Susceptible Locations in Medical Records via Adversarial Attacks on Deep Predictive Models. In _KDD_.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _CVPR_.
  * Tang et al. (2018) Fengyi Tang, Cao Xiao, Fei Wang, and Jiayu Zhou. 2018\.  Predictive modeling in urgent care: a comparative study of machine learning approaches.  _JAMIA Open_ (2018).
  * Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt. 1998.  Learning to learn: Introduction and overview.  In _Learning to learn_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NIPS_.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016\.  Matching networks for one shot learning. In _NIPS_.
  * Wang et al. (2012) Fei Wang, Noah Lee, Jianying Hu, Jimeng Sun, and Shahram Ebadollahi. 2012.  Towards heterogeneous temporal clinical event pattern discovery: a convolutional approach. In _KDD_.
  * Young et al. (2018) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018.  Recent trends in deep learning based natural language processing.  _IEEE Computational Intelligence Magazine_ 13, 3 (2018).


# An Exploratory Study on Long Dialogue Summarization:
What Works and What’s Next

Yusen Zhang∗♣ Ansong Ni∗† Tao Yu† Rui Zhang♣
Chenguang Zhu♢ Budhaditya Deb♢ Asli Celikyilmaz‡
Ahmed Hassan Awadallah ♢ Dragomir Radev†
♣Penn State University †Yale University ♢Microsoft Research ‡ Facebook AI
Research
{yfz5488<EMAIL_ADDRESS><EMAIL_ADDRESS>
{ansong.ni, tao.yu<EMAIL_ADDRESS>
{chezhu, Budha.Deb<EMAIL_ADDRESS>

###### Abstract

Dialogue summarization helps readers capture salient information from long
conversations in meetings, interviews, and TV series. However, real-world
dialogues pose a great challenge to current summarization models, as the
dialogue length typically exceeds the input limits imposed by recent
transformer-based pretrained models, and the interactive nature of dialogues
makes relevant information more context-dependent and sparsely distributed
than news articles. In this work, we perform a comprehensive study on long
dialogue summarization by investigating three strategies to deal with the
lengthy input problem and locate relevant information: (1) extended
transformer models such as Longformer, (2) retrieve-then-summarize pipeline
models with several dialogue utterance retrieval methods, and (3) hierarchical
dialogue encoding models such as HMNet. Our experimental results on three long
dialogue datasets (QMSum, MediaSum, SummScreen) show that the retrieve-then-
summarize pipeline models yield the best performance. We also demonstrate that
the summary quality can be further improved with a stronger retrieval model
and pretraining on proper external summarization datasets.

††∗Equal Contribution.††‡The work was done when Asli was at MSR.

##  1 Introduction

Large amount of dialogue data have been produced in meetings, TV series, and
interviews Chen et al. (2021); Zhong et al. (2021); Zhu et al. (2021).
Dialogue summarization aims to generate a short summary for long dialogues to
help the readers capture important information more efficiently.

A number of existing works on dialogue summarization focus on extracting the
main events of a short conversation Gliwa et al. (2019); Rohde et al. (2021).
However, unlike the short dialogues which contains less than 20 utterances,
some tasks for summarizing much longer dialogues have been proposed recently
Chen et al. (2021); Zhong et al. (2021). These datasets are usually derived
from meetings and interviews, with hundreds of turns in each dialogue. The
length of such dialogues typically exceeds the input limits imposed by recent
transformer-based models Lewis et al. (2020), making it difficult to train an
end-to-end summarization model for such tasks. This poses the challenge: How
can we effectively use the current neural summarization models on dialogues
that greatly exceed their length limits?

Additionally, compared with document summarization, dialogues are interactive
in nature, makes it more context-dependent and the information in dialogues is
more sparsely distributed. Besides, the informal language used in dialogues
leads to difficulties in modeling relevance and salience. To solve these
issues, hierarchical methods are proposed to model the dialogues at turn level
Zhu et al. (2020a); Rohde et al. (2021). However, generating a short summary
that contains all the salient information remains challenging.

In this paper, we systematically investigate these issues on dialog
summarization: we first explore the various solutions to the lengthy input
problem. Then, we analyze and compare the methods to improve generic
summarization models on challenging dialogue datasets. To address the long
input issue, we investigate extended transformer models such as Longformer
Beltagy et al. (2020), and several dialogue utterance retrieval methods for a
retrieve-then-summarize pipeline model, as well as hierarchical dialogue
encoding models. For the specific challenges in dialogues, we explore
different datasets for pretraining to test the transferability between similar
summarization tasks. We evaluate these models on three recent long dialogue
summarization datasets: QMSum for meetings Zhong et al. (2021), MediaSum for
interviews Zhu et al. (2021), SummScreen for TV series transcripts Chen et al.
(2021). In our experiments, we find that the pipeline method with a dialogue
utterance retrieval model yields the best performance, and it can be further
improved with a stronger retrieval model. Our experiment results also suggest
that pretraining on proper external summarization datasets can effectively
improve the performance of dialogue summarization models.

##  2 Related Work

##### Long Sequence Summarization

Recent summarization models are based on Transformer (Vaswani et al., 2017)
that has a quadratic time and memory complexity with respect to the input
length, preventing it from being used for longer sequences. To address this
issue, Beltagy et al. (2020) used the sliding window and global attention,
while Zaheer et al. (2020) used a combination of random, sliding window and
global attention mechanism to reduce the quadratic complexity to close-linear.
Previous benchmarks for long sequence summarization mostly focus on documents
instead of dialogues: PubMed and Arxiv Cohan et al. (2018) consists of
scientific papers which are typically very long; BillSum (Kornilova and
Eidelman, 2019) is a corpus of U.S. Congressional bills and their summaries;
BigPatent (Sharma et al., 2019) contains 1.3 million U.S. patent files and
human-written summaries.

##### Dialogue Summarization

Dialogue summarization aims to generate concise summaries for dialogues, such
as meetings (McCowan et al., 2005; Janin et al., 2003; Zhong et al., 2021;
Shang et al., 2018; Zhu et al., 2020a), TV series (Chen et al., 2021),
interviews (Zhu et al., 2021), and chit-chat (Gliwa et al., 2019; Zhao et al.,
2020; Chen and Yang, 2021). Some summarization datasets (not limited to
dialogues) contain queries asking for summarizing specific parts of dialogues
(Zhong et al., 2021; Nema et al., 2017), while others only need to summarize
whole dialogues (Chen et al., 2021; Gliwa et al., 2019; Hermann et al., 2015).
As for dialogue summarization models, Zhu et al. (2020b) described a
hierarchical model for both inner- and cross-utterance attention, while Chen
and Yang (2020) proposed a multi-view decoder to leverage different extracted
views of dialogues, such as topic view and stage view.

| QMSum | SummScreen | MediaSum*
---|---|---|---
Source | Meeting | TV Series | Interviews
Query-based | YES | NO | NO
# examples | 1.8k | 26.9k | 463.6k
# input tokens | 9069.8 | 6612.5 | 1553.7
# summary tokens | 69.6 | 337.4 | 14.4
# speakers | 9.2 | 28.3 | 6.5

Table 1: Comparison between three long dialogue summarization datasets we
mainly study in this work. Numbers in the table are averaged across all
samples. (*: MediaSum is only used for pretraining)

##  3 Methodology

In this section, we will introduce the dataset used to evaluate and pretrain
the model, two types of summary models, and the details of the experiment
setup.

###  3.1 Datasets

To explore the problems in long dialogue summarization, we leverage three
different long dialogue summarization tasks as main datasets:
QMSum (Zhong et al., 2021) is a query-based multi-domain meeting summarization
dataset annotated by humans. It contains 1,808 queries together with 232 long
meeting transcripts, with topics as software product, academics, and committee
meetings. QMSum also contains annotated gold spans which could be used as the
gold labels for training the retrievers;
MediaSum (Zhu et al., 2021) is a large-scale media interview dataset
consisting of 463.6K transcripts collected from NPR and CNN. Because MediaSum
contains short summaries, i.e. only a short sentence representing the topic,
we only use this dataset for pretraining and analysis. Due to the huge size of
this dataset, 20k samples are randomly extracted for pretraining;
SummScreen (Chen et al., 2021) is a dialogue summarization dataset consisting
of 26.9k pairs of TV series transcripts and human-annotated summaries. It
comes with two sources for recaps, and in this work, we choose one of
them,i.e. “Forever Dreaming”, for which we call SummScreen-FD as our
benchmark.

Tab. 1 shows the statistics for these three long dialogue datasets.
Additionally, we also consider CNN/Dailymail (Hermann et al., 2015) (CNN/DM),
XSum (Narayan et al., 2018), and SAMSum (Gliwa et al., 2019) as datasets for
pretraining in our experiments.

###  3.2 Models

####  3.2.1 Retrieve-then-summarize Pipeline

Dialogues tend to be relatively long, and most existing summarization models
cannot process such long inputs. The two-stage retrieve-then-summarize
pipeline first retrieves the most relevant subtext in the dialogue and then
feeds to a summarizer. We experiment with the following retrievers:

  * •

TF-IDF (Jones, 1972) Based on bag-of-words representation, TF-IDF measuers
term frequency (TF) and normalizes them with inversed document frequency
(IDF);

  * •

BM25 (Robertson and Zaragoza, 2009) Similar to TF-IDF but accounts for
document length and term saturation;

  * •

Locator 111We obtained the locator output from the original authors. The
utterance locator model proposed by Zhong et al. (2021) using convolution
neural networks with BERT (Devlin et al., 2019).

For TF-IDF and BM25, we limit the number of retrieved utterances to be at most
10% of the whole dialogue, while we directly use the utterances predictor by
Locator in its setting. After retrieval, we use the BART-large model fine-
tuned on the output of the various retrievers to produce the summary.

####  3.2.2 End-to-end Summarization Models

To study how current state-of-the-art neural summarizers perform on long
dialogue summarization, we choose the following three models:

BART (Lewis et al., 2020) is a transformer-based encoder-decoder model which
obtains a number of state-of-the-art results on various text generation tasks.
We use this model as our baseline summarization model for studying its
ablations under different settings. The maximum number of input tokens is
1,024 so we truncate the input when it exceeds such limit.222We also tried to
extend the positional embeddings to 2,048 for BART to accept longer input but
found the results to be worse in our case.

HMNet (Zhu et al., 2020a) is a hierarchical network for dialogue
summarization. It models the structure of the dialogue, using a token level
encoder to encode each sentence and a turn level encoder for aggregating each
turn. We use HMNet as a representative for the hierarchical type of models and
compare it with other baselines. Due to the limitation of the memory cost, we
constrain the maximum number of tokens to be 8,192 for HMNet, which is 8x as
large as BART mentioned above.

Longformer (Beltagy et al., 2020) adapts the self-attention mechanism from
full attention matrix to sliding window attention + global attention, which is
more memory efficient. Longformer can accept up to 16K tokens and has shown
improvement over long document summarization using its long-encoder-decoder
(LED) variant. We allow the maximum input of 4,096 tokens for Longformer and
cutoff the rest of the input, as we found further increasing such limit yields
no improvements.

To incorporate queries in QMSum for these end-to-end models, we simply append
the queries to the front of the meeting transcripts, as it is a standard
practice for query-based summarization and also question answering (Devlin et
al., 2019).

###  3.3 Experiment Setup

For a fair comparison between all models, we fit all of the models into the
same RTX 8000 GPU with 48 GiB of GPU memory. We adopt the
fairseq333https://github.com/pytorch/fairseq implementation for BART, and the
original code base for both Longformer444https://github.com/allenai/longformer
and HMNet555https://github.com/microsoft/HMNet. We inherit the hyperparameters
for all those models for fine-tuning in our experiments.666For more
implementation details, please refer to our experiment code:
https://github.com/chatc/LongDialSumm. Our most expensive experiments are
fine-tuning for HMNet and Longformer, which take around 8 hours, while the
runtime for BART model is less than one hour. We use ROUGE Lin (2004) as our
main evaluation metric and pyrouge
library777https://github.com/bheinzerling/pyrouge as the ROUGE implementation
throughout all experiments.

##  4 Result and Analysis

Here we demonstrate our findings in four corresponding subsections. We also
show some concrete examples and perform qualitative analysis in § 4.5

###  4.1 Dealing with Long Dialogues

We compare several methods for addressing the long input issue for dialogue
summarization, including different utterance retrieval methods describe in §
3.2.1 for a retrieve-then-summarize framework, heuristics for shortening the
dialogue as well as baseline methods to establish reasonable bounds. From Tab.
2, we can see that even in the query-based dialogue summarization with QMSum,
randomly selecting utterances still presents a strong baseline. Over different
modeling choices, the retrieve-then-summarize framework generally works better
than end-to-end learning with dialogue cutoff at maximum input length. We do
not observe an advantage of using Longformer over the BART model. This raises
the question on whether all utterances in the dialogue are needed to produce a
good summary or irrelevant utterances would add more noise. Moreover, we
notice that all these methods present a non-trivial gap with the summarization
performance on the gold span, which uses relevant utterances annotated by
humans. This suggests that there is plenty room for improvement if a better
utterance retrieval method is developed.

Methods | R-1 | R-2 | R-L
---|---|---|---
Retrieve-then-summarize |  |
Random | 31.1 | 7.9 | 20.9
TF-IDF | 32.5 | 8.5 | 21.4
BM25 | 32.9 | 9.0 | 22.0
Locator | 29.9 | 7.6 | 19.6
Gold span | 36.6 | 14.0 | 25.5
End-to-end (Cutoff at max # tokens) |  |
BART-large(1024) | 32.6 | 8.7 | 21.6
Longformer-large(4096) | 31.6 | 7.8 | 20.5
Table 2: Comparison of different methods for addressing the length of the dialogues on QMSum. All "retrieve-then-summarize" pipelines use BART-Large as a backend for summarization. "Gold span" denotes the annotated relevant turns in QMSum.  | QMSum | SummScreen-FD
---|---|---
| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-1 | ROUGE-2 | ROUGE-L
BART-Large | 36.56 | 14.05 | 25.54 | 27.12 | 4.88 | 16.82
\+ XSum | 34.90 | 13.49 | 24.90 | 27.17 | 4.59 | 17.02
\+ MediaSum | 34.23 | 13.06 | 25.21 | 27.73 | 5.03 | 17.09
\+ CNN/DM | 39.88 | 15.94 | 28.02 | 28.86 | 5.55 | 17.39
\+ CNN/DM-SAMSum | 35.46 | 12.52 | 24.62 | 28.15 | 5.41 | 17.25
\+ CNN/DM-MediaSum | 36.79 | 13.69 | 25.94 | 28.68 | 5.31 | 17.42
Table 3: The performance of BART-large models that are pretrained on various
summarization datasets.

###  4.2 Robustness to Input Length

As we discussed, some dialogues (e.g., QMSum) contain more than 20k tokens.
They exceed the input limitation of most existing summarization models. In
this section, we further analyze the performance of summarization models as
the input length changes. To compare the robustness between two types of
models (mainly BART and HMNet), we divide the test dialogues by the number of
tokens. As we can see in Fig. 1, the performance of the BART model decreases
sharply when the dialogue input becomes longer while the HMNet shows the
opposite effect. This could be the result of their unique properties: BART is
pretrained on the datasets with a limited length (i.e., 1,024) and the input
has to be truncated to fit the limitation, while HMNet obtains more
information when the input is longer. However, the overall performance of
HMNet is worst than BART.

| ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
BART-CNN |  |  |
w/o Query | 34.48 | 11.5 | 23.11
w/ Query | 39.88 | 15.94 | 28.02
HMNet |  |  |
w/o Query | 35.1 | 10.1 | 30.8
w/ Query | 36.8 | 10.9 | 31.9
Table 4: The performance comparison between BART and HMNet models on the
query-based meeting summarization QMSum dataset.  Figure 1: The ROUGE-L F1
scores of BART and HMNet on QMSum stratified by the number of input dialgue
tokens.

###  4.3 Incorporating Queries

Certain dialogue summarization tasks, such as QMSum, require generating a
summary based on a specific question about the dialogue (e.g., opinion of a
speaker or conclusion to a topic). In this section, we study the influence of
incorporating queries in dialogue summarization. Tab. 4 shows the performance
of two models, BART and HMNet, on QMSum with and without queries at the
beginning of the input. For the input to the two models, we use the gold
relevant text spans given a query in QMSum to avoid the influences of
retrieval models. The results show that encoding queries has a large impact on
both types of models, especially for BART, even if the gold utterances are
given.

###  4.4 Transfer Ability between Different Tasks

Pretraining has been shown effective for document summarization by introducing
external knowledge from other similar tasks (Hermann et al., 2015; Fabbri et
al., 2019). We hypothesize that it is especially important for dialogue
summarization because the dataset size is usually small. Therefore, we study
the transfer learning between different dialogue summarization tasks via
pretraining. Tab. 3 shows the performance of BART-large models that are
pretrained using different datasets and later fine-tuned on QMSum and
SummScreen-FD. The results show that BART-large pretrained on CNN/Dailymail
dataset (BART-CNN) yields the best performance after finetuning, though
CNN/Dailymail consists of News articles and is not in dialogue format. We also
note that pretraining on external datasets can also hurt the performance, and
thus such pretraining datasets need to be carefully chosen.

We also analyze the performance of BART-large by pretraining it on more than
one dataset to test if BART-large can be further improved. We use the BART-
large model pretrained on CNN/DM (BART-CNN) as baseline model since BART-CNN
yields the best performance compared with the others. And then pretrain the
same BART-CNN model on SAMSum and MediaSum separately. However, Tab. 3 shows
that after pretraining BART-CNN on these two datasets, ROUGE scores decrease
sharply on QMSum dataset, and lightly on SummScreen-FD dataset except for
ROUGE-L. This result demonstrates that pretraining on multiple dataset may not
further improve the performance of the pretrained models.

###  4.5 Case Study

We exam several summaries generated by BART-large model pretrained on three
different datasets. We found that the BART-CNN model yields the best output
with the least number of syntax errors and the closest content to the desired
ones, while the output of BART-MediaSum model is usually shorter than Gold
resulting in incomplete generation, and BART-XSum model usually predicts
summaries with errors and duplication. This could be the result of data bias
of pretraining datasets — Summaries in MediaSum and XSum are shorter than
CNN/DM. However, despite the better performance of BART-CNN model, these cut-
off models fail to predict some part of the gold summary when the number of
tokens in input dialogue is larger than the maximum input length of the model.
For concrete examples, please refer to Appendix A.

##  5 Conclusion and Future Work

We first explore the lengthy input problem of dialogue summarization through
experiments on transformers and retrieval models. We conclude that the
retrieve-summarize pipeline results in the best performance. Then, the
experiments demonstrate the important role of queries and robustness to input
length for different types of models. We found that adding a single query
sentence in the input greatly improves ROUGE scores on QMSum. Additionally,
BART performs worse when the input is beyond 512 tokens, even with extended
positional embeddings; on the contrary, the hierarchical model performs better
for longer inputs. We also test the transferability of summarization datasets
by pretraining the language model on similar tasks. We conclude that the BART-
large model pretrained on CNN/DM yields the best performance on both QMSum and
SummScreen-FD.

For future work on solving the long input problem, we found that using an
utterance retrieval model for summarization is a promising direction, yet
modeling relevance between query and dialogue utterances remains a challenging
task. And for the summarization models, it is worth exploring methods to 1)
pretrain on valuable datasets for dialogue summarization, 2) better fuse the
queries into the neural models, and 3) make the model robust to the input
length (like HMNet) and maintain the high performance in the meantime (like
BART).

## Acknowledgments

The authors would like to thank Ming Zhong, Da Yin, Yang Liu for their
discussions and anonymous reviewers for their helpful comments. This work is
supported in part by a grant from Microsoft Research.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _ArXiv preprint_ , abs/2004.05150.
  * Chen and Yang (2020) Jiaao Chen and Diyi Yang. 2020.  Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4106–4118, Online. Association for Computational Linguistics.
  * Chen and Yang (2021) Jiaao Chen and Diyi Yang. 2021.  Structure-aware abstractive conversation summarization via discourse and action graphs.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1380–1391, Online. Association for Computational Linguistics.
  * Chen et al. (2021) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021.  Summscreen: A dataset for abstractive screenplay summarization.  _ArXiv preprint_ , abs/2104.07091.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Janin et al. (2003) Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003.  The icsi meeting corpus.  In _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)._ , volume 1, pages I–I. IEEE.
  * Jones (1972) Karen Sparck Jones. 1972.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of documentation_.
  * Kornilova and Eidelman (2019) Anastassia Kornilova and Vladimir Eidelman. 2019.  BillSum: A corpus for automatic summarization of US legislation.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 48–56, Hong Kong, China. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * McCowan et al. (2005) Iain McCowan, Jean Carletta, Wessel Kraaij, Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas Hain, J Kadlec, Vasilis Karaiskos, et al. 2005.  The ami meeting corpus.  In _Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research_ , volume 88, page 100. Citeseer.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Nema et al. (2017) Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017.  Diversity driven attention model for query-based abstractive summarization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1063–1072, Vancouver, Canada. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends in Information Retrieval_ , 3(4):333–389.
  * Rohde et al. (2021) Tobias Rohde, Xiaoxia Wu, and Yinhan Liu. 2021.  Hierarchical learning for generation with long source sequences.  _ArXiv preprint_ , abs/2104.07545.
  * Shang et al. (2018) Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2018.  Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 664–674, Melbourne, Australia. Association for Computational Linguistics.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  BIGPATENT: A large-scale dataset for abstractive and coherent summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2204–2213, Florence, Italy. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020\.  Big bird: Transformers for longer sequences.  _ArXiv preprint_ , abs/2007.14062.
  * Zhao et al. (2020) Lulu Zhao, Weiran Xu, and Jun Guo. 2020.  Improving abstractive dialogue summarization with graph structures and topic words.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 437–449, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021.  QMSum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5905–5921, Online. Association for Computational Linguistics.
  * Zhu et al. (2021) Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021.  MediaSum: A large-scale media interview dataset for dialogue summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5927–5934, Online. Association for Computational Linguistics.
  * Zhu et al. (2020a) Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020a.  A hierarchical network for abstractive meeting summarization with cross-domain pretraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 194–203, Online. Association for Computational Linguistics.
  * Zhu et al. (2020b) Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020b.  A hierarchical network for abstractive meeting summarization with cross-domain pretraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 194–203, Online. Association for Computational Linguistics.

##  Appendix A Generated Cases

Tab. 5 shows some concrete sample summaries generated by BART-large model
pretrained on various datasets.

|  QMSum |  SummScreen-FD
---|---|---
MediaSum |  Transcribers are working on transcribing the data from the corpus . The next step is to insure that the data are clean first , and then channelized . The transcribers are also working on ensuring that the mark-up is consistent all the way throughout . |  Sydney and Will are sent to a secret CIA project to find out if their father is alive or dead . Meanwhile , Sydney and Vaughn are sent to a secret CIA facility to find out what Irina is up to .
XSum |  The transcribers have transcribed about thirty-five hours of transcripts from the corpus . The next step is to insure that the data is clean first , and then channelized . The transcribers are working on is to insure that the data is clean first , and then channelized . The transcribers are also incorporating additional conventions that Liz requested in terms of having a systematic handling of numbers , acronyms and acronyms which I had n’t been specific about . |  Sydney and Will are shocked to learn that Sydney’s father , who was killed in Madagascar , is alive and working for the CIA . Will is also shocked to learn that Sydney ’s mother , who was killed in the Rambaldi experiment , is alive . Will is also shocked to learn that Sydney ’s father is a scientist . Will is also shocked to learn that Sydney ’s mother is a scientist . Will is also shocked to learn that Sydney ’s mother is a scientist . $\cdots$
CNN |  The team was working on transcribing the data , and the next step was to ensure that the data was clean first , and then channelized . The team was working on ensuring that the data was spell-checked , that the mark-up was consistent all the way throughout , and that they incorporated additional conventions that Liz requested in terms of having a systematic handling of numbers , acronyms , and acronyms which they had n’t been specific about . |  Sydney and Will investigate the death of her father , who was killed in a Russian KGB operation in 1982 . They discover that the Rambaldi device was a Russian spy device , which was used to test the IQ of children . Sydney ’s father was a KGB agent , and she is now a KGB agent . She is also a double agent , and she is working for the CIA . She is also working for the CIA to find out who is behind the death of her father . Meanwhile , Irina is worried about her father ’s death , and she is worried about her relationship with Vaughn .
Gold |  Efforts by speaker fe008 are in progress to ensure that transcripts are clean ( i.e . spell checked ) , channelized , and conform to set conventions regarding the coding of numbers , acronyms , and explicit comments ( e.g . door slams , coughs , and laughter ) . Subsequent efforts by speaker fe008 will be to tighten up boundaries on the time bins . Inter-annotator agreement was reported to be very good .Speaker mn014 ’s multi-channel speech/non-speech segmenter is in use . |  Sydney races to find a cure for Vaughn , but in order to find the antidote , Sydney must make a deal with Sark that could endanger Sloane ’s life . Meanwhile , Will continues his research for Vaughn and discovers some disturbing inconsistencies involving 20-year - old standardized IQ tests . Sydney finds out that Vaughn has a girlfriend .

Table 5: Sample output summaries of various pretrained models on QMSum and
SummScreen. The summary $S$ of row $X$, column $Y$ indicates that BART-large
model which is pretrained on $X$ dataset generates summary $S$ from test set
of $Y$. The errors and duplication are marked in red. The out-of-boundary
contents are marked in grey. Tokens marked in brown indicate the keywords
emerged in Gold summary.