# Learning Contextualized Document Representations for Healthcare Answer
Retrieval

Sebastian Arnold<EMAIL_ADDRESS>0000-0001-7788-6100 Beuth
University of Applied SciencesLuxemburger Strasse 1013353BerlinGermany ,
Betty van Aken<EMAIL_ADDRESS>Beuth University of Applied
SciencesLuxemburger Strasse 1013353BerlinGermany ,  Paul Grundmann
<EMAIL_ADDRESS>Beuth University of Applied SciencesLuxemburger
Strasse 1013353BerlinGermany ,  Felix A. Gers<EMAIL_ADDRESS>Beuth
University of Applied SciencesLuxemburger Strasse 1013353BerlinGermany and
Alexander Löser<EMAIL_ADDRESS>Beuth University of Applied
SciencesLuxemburger Strasse 1013353BerlinGermany

(2020)

###### Abstract.

We present Contextual Discourse Vectors (CDV), a distributed document
representation for efficient answer retrieval from long healthcare documents.
Our approach is based on structured query tuples of entities and aspects from
free text and medical taxonomies. Our model leverages a dual encoder
architecture with hierarchical LSTM layers and multi-task training to encode
the position of clinical entities and aspects alongside the document
discourse. We use our continuous representations to resolve queries with short
latency using approximate nearest neighbor search on sentence level. We apply
the CDV model for retrieving coherent answer passages from nine English public
health resources from the Web, addressing both patients and medical
professionals. Because there is no end-to-end training data available for all
application scenarios, we train our model with self-supervised data from
Wikipedia. We show that our generalized model significantly outperforms
several state-of-the-art baselines for healthcare passage ranking and is able
to adapt to heterogeneous domains without additional fine-tuning.

document representation, passage ranking, discourse vectors

††copyright: iw3c2w3††journalyear: 2020††doi:
10.1145/3366423.3380208††conference: Proceedings of The Web Conference 2020;
April 20–24, 2020; Taipei, Taiwan††booktitle: Proceedings of The Web
Conference 2020 (WWW ’20), April 20–24, 2020, Taipei, Taiwan††isbn:
978-1-4503-7023-3/20/04††submissionid: 1201††ccs: Information systems Document
representation††ccs: Information systems Information retrieval††ccs:
Information systems Decision support systems

##  1\. Introduction

[The query Q=(IgA nephropathy, symptoms) and a snippet of text from
Wikipedia.]The query Q=(IgA nephropathy, symptoms) and a snippet of text taken
from the Wikipedia article about IgA nephropathy, where a long answer passage
is highlighted.

Figure 1. Example of a structured entity/aspect query $Q$ and a highlighted
answer passage from Wikipedia. Note that the answer is part of a longer
document and there is almost no word overlap between query and answer passage.

In a clinical decision support system (CDSS), doctors and healthcare
professionals require access to information from heterogeneous sources, such
as research papers (Gorman et al., 1994; Schardt et al., 2007), electronic
health records (Hanauer et al., 2015), clinical case reports (Fujiwara et al.,
2018), reference works and knowledge base articles. Differential diagnosis is
an important task where a doctor seeks to retrieve answers for non-factoid
queries about diseases, such as “symptoms of IgA nephropathy” (see Figure 1).
A relevant answer typically spans multiple sentences and is most likely
embedded into the discourse of a long document (Yang et al., 2016b; Cohan et
al., 2018).

Evidence-based medicine (EBM) has made efforts to structure physicians’
information needs into short structured question representations, such as PICO
(patient, intervention, comparison, outcome) (Richardson et al., 1995)
and—more general—well-formed background / foreground questions (Cheng, 2004).
We support this important query intention and define a query as structured
tuple of _entity_ (e.g. a disease or health problem) and _aspect_. Our model
is focused on clinical aspects such as therapy, diagnosis, etiology,
prognosis, and others, which have been described in the literature previously
by manual clustering of semantic question types (Huang et al., 2006) or
crawling medical Wikipedia section headings (Arnold et al., 2019). In a CDSS,
a doctor can express these query terms with identifiers from a knowledge base
or medical taxonomy, e.g. UMLS, ICD-10 or Wikidata. The system will support
the user in assigning these links by search and auto-completion operators
(Schneider et al., 2018; Fujiwara et al., 2018), which allows us to use these
representations as input for the answer retrieval task.

Several methods have been proposed to apply deep neural networks for effective
information retrieval (Guo et al., 2016; Mitra et al., 2017; Dai et al., 2018)
and question answering (Seo et al., 2017; Wang et al., 2017), also with focus
on healthcare (Zhu et al., 2019; Jin et al., 2019). However, our CDSS scenario
poses a unique combination of open challenges to a retrieval system:

  1. (1)

_Task coverage:_ Query intentions span a broad range in specificy and
complexity (Huang et al., 2006; Nanni et al., 2017). For example, medical
specialists may pose very precise queries that align with a pre-defined
taxonomy and focus on rare diseases. On the other hand, nursing staff might
have broader and more heterogeneous questions. However, in most cases we do
not have access to task-specific training data, so training the model for a
single intention is not feasible. We therefore require a generalized query
representation that covers a broad range of intents and taxonomies, even with
limited training data.

  2. (2)

_Domain adaptability:_ In many cases we do not even have textual data readily
available at training time from all resources in a CDSS. However, we observe
linguistic and semantic shifts between the heterogeneous types of text, e.g.
different use of terms and abbreviations among groups of doctors. Therefore we
face a zero-shot retrieval task that requires robust domain transfer abilities
across diverse biomedical, clinical and healthcare text resources (Logeswaran
et al., 2019).

  3. (3)

_Contextual coherence:_ Answers are often expressed as passages in context of
a long document. Therefore the model needs to respect long-range dependencies
such as the sequence of micro-topics that establish a coherent ‘train of
thought’ in a document (Arora et al., 2016; Arnold et al., 2019). At the same
time, the model is required to operate on a fine granularity (e.g., on
sentence level) rather than on entire documents to be able to capture the
boundaries of answers (Keikha et al., 2014).

  4. (4)

_Efficient neural information retrieval:_ Finally, all documents in the CDSS
need to be accessible with fast ad-hoc queries by the users. Many question
answering models are based on pairwise similarity, which is computationally
too intensive when applied to large-scale retrieval tasks (Gillick et al.,
2018). Instead, we require a continuous retrieval model that allows for
offline indexing and approximate nearest neighbor search with high recall
(Gillick et al., 2018), even for rare queries and with low latency in the
order of milliseconds.

We approach these challenges and present _Contextual Discourse Vectors_
(CDV)111Code and evaluation data is available at
https://github.com/sebastianarnold/cdv, a neural document representation which
is based on discourse modeling and fulfills the above requirements. Our method
is the first to address answer retrieval with structured queries on long
heterogeneous documents from the healthcare domain.

CDV is based on hierarchical layers to encode word, sentence and document
context with bidirectional Long Short-Term Memory (BLSTM). The model uses
multi-task learning (Caruana, 1997) to align the sequence of sentences in a
long document to the clinical knowledge encoded in pre-trained entity and
aspect vector spaces. We use a dual encoder architecture (Gillick et al.,
2018), which allows us to precompute discourse vectors for all documents and
later answer ad-hoc queries over that corpus with short latency (Gillick et
al., 2019). Consequently, the model predicts similarity scores with sentence
granularity and does not require an extra inference step after the initial
document indexing.

We apply our CDV model for retrieving passages from various public health
resources on the Web, including NIH documents and Patient articles, with
structured clinical query intentions of the form
$\langle\text{entity},\text{aspect}\rangle$. Because there is no training data
available from most sources, we use a self-supervised approach to train a
generalized model from medical Wikipedia texts. We apply this model to the
texts in our evaluation in a zero-shot approach (Palatucci et al., 2009)
without additional fine tuning.

In summary, the major contributions of this paper include:

  * •

We propose a structured entity/aspect healthcare query model to support the
essential query intentions of medical professionals. Our task is focused on
the efficient retrieval of answer passages from long documents of
heterogeneous health resources.

  * •

We introduce CDV, a contextualized document representation for passage
retrieval. Our model leverages a dual encoder architecture with BLSTM layers
and multi-task training to encode the position of discourse topics alongside
the document. We use the representations to answer queries using nearest
neighbor search on sentence level.

  * •

Our model utilizes generalized language models and aligns them with clinical
knowledge from medical taxonomies, e.g. pre-trained entity and aspect
embeddings. Therefore, it can be trained with sparse self-supervised training
data, e.g. from Wikipedia texts, and is applicable to a broad range of texts.

  * •

We prove the applicability of our CDV model with extensive experiments and a
qualitative error analysis on nine heterogeneous healthcare resources. We
provide additional entity/aspect labels for all datasets. Our model
significantly outperforms existing document matching methods in the retrieval
task and can adapt to different healthcare domains without fine-tuning.

In this paper, we first give an overview of related research (Section 2).
Next, we introduce our query representation (Section 3). Then, we focus on the
contextual document representation model (Section 4). Finally, we discuss the
findings of our experimental evaluation of the healthcare retrieval task
(Section 5) and summarize our conclusions (Section 6).

##  2\. Background

There is a large amount of work on _question answering_ (QA) (Seo et al.,
2017; Wang et al., 2017), also applied to healthcare (Abacha et al., 2019; Jin
et al., 2019) which focuses primarily on factoid questions with short answers.
Typically, these models are trained with labeled question-answer pairs.
However, it was shown that these models are not suitable for extracting local
aspects from long documents, and especially not for open-ended, long answer
passages (Tellex et al., 2003; Keikha et al., 2014; Yang et al., 2016b; Zhu et
al., 2019). We therefore frame our task as a _passage retrieval_ problem,
where the system’s goal is to extract a concise snippet (typically 5–20
sentences) out of a large number of long documents. Furthermore, following
studies from EBM (Richardson et al., 1995; Cheng, 2004; Huang et al., 2006),
we focus on _structured healthcare queries_ instead of free-text questions.

##### Discourse-aware representations

Recently, new approaches have emerged that represent local information in the
context of long documents. For example, Cohan et al. (2018) approach the
problem as abstractive summarization task. The authors use hierarchical
encoders to model the discourse structure of a document and generate summaries
using an attentive discourse-aware decoder. In our prior work on SECTOR
(Arnold et al., 2019), we apply a segmentation and classification method to
long documents to identify coherent passages and classify them into 27
clinical aspects. The model produces a continuous topic embedding on sentence
level using BLSTMs, which has similar properties to the micro-topics described
earlier by Arora et al. (2016) as _discourse vector_ (“what is being talked
about”).

We follow these ideas as the groundwork for our approach. Our proposed model
is based on a hierarchical architecture to encode a continuous discourse
representation. To the best of our knowledge, our model is the first to use
discourse-aware representations for answer retrieval. Additionally, we address
the problem of sparse training data and propose a multi-task approach for
training the model with self-supervised data instead of labeled examples.

##### Passage matching

A baseline approach to the passage retrieval problem is to split longer
documents into individual passages and rank them independently according to
their relevance for the query. Passage matching has been done using term-based
methods (Robertson and Jones, 1976; Salton and Buckley, 1988), most
prominently in TF-IDF (Jones, 1972) or Okapi BM25 (Robertson et al., 1995).
However, these methods usually do not perform well on long passages or when
there is minimal word overlap between passage and query. Therefore, most
neural matching models tackle this vocabulary mismatch using semantic vector-
space representations.

Representation-based matching models aim to match the continuous
representations of queries and passages using a similarity function, e.g.
cosine distance. This can be done on sentence level (ARC-I (Hu et al., 2014)),
which does not work well if queries are short and passages are longer than a
few sentences. Therefore, most approaches learn distinct query and passage
representations using feed-forward (DSSM (Huang et al., 2013)) or CNN
convolutional neural networks (C-DSSM (Shen et al., 2014)).

Interaction-based matching models focus on the complex interaction between
query and passage. These models use CNNs on sentence level (ARC-II (Hu et al.,
2014)), match query terms and words using word count histograms (DRMM (Guo et
al., 2016)), word-level dot product similarity (MatchPyramid (Pang et al.,
2016)), attention-based neural networks (aNMM (Yang et al., 2016a)), kernel
pooling (K-NRM (Xiong et al., 2017)) or convolutional n-gram kernel pooling
(Conv-KNRM (Dai et al., 2018)). Eventually, Zhu et al. (2019) utilize
hierarchical attention on word and sentence level (HAR) to capture interaction
of the query with local context in long passages.

While interaction-based models can capture complex correlations between query
and passage, these models do not include contextualized local information—e.g.
long-range document context that comes before or after a passage—which might
contain important information for the query. To overcome this problem, Mitra
et al. (2017) combine document-level representations with interaction features
in a deep CNN model (Duet). Wan et al. (2016) utilize BLSTMs (MVLSTM) to
generate positional sentence representations across the entire document.

We combine the representation approach with interaction. Our proposed model is
able to learn the interaction between the words of the passage and the
discourse using a hierarchical architecture. At the same time, it encodes
fixed sentence representations that we use to match query representations.
Consequently, our model does not require pairwise inference between all
query–sentence pairs, which is usually circumvented by re-ranking candidates
(Gillick et al., 2018). Instead, our model requires only a single pass through
all documents at index time. Furthermore, by encoding discourse-aware
representations, the model is able to access long-range document context which
is normally hidden after the passage split. We compare our approach to all the
discussed matching models and review these properties again in Section 5.

##  3\. Query Model

Our first challenge is to design a query model which can adapt to a broad
number of healthcare answer retrieval tasks and utilizes the information
sources available in a CDSS. In this section, we introduce a vector-space
representation for this purpose.

We define a query as a structured tuple
$Q=\langle\text{entity},\text{aspect}\rangle$. This approach of using two
complementary query arguments originates from the idea of structured
background/foreground questions in EBM (Cheng, 2004) and has been used before
in many triple-based retrieval systems (Adolphs et al., 2011). In our
healthcare scenario, we restrict entities to be of type _disease_ , e.g. “IgA
nephropathy”, and aspects from the clinical domain, e.g. “symptoms”,
“treatment”, or “prognosis”. We discuss these two spaces in Sections 3.1 and
Section 3.2 and propose their combination in Section 3.3. In general, our
model is not limited to the query spaces used in this paper and further
extendable to a larger number of arguments.

###  3.1. Entity Space

The first part of our problem is to represent the entity in focus of a query.
In contrast to interaction-based models, which are applied to query–document
pairs, our approach is to decouple entity encoding and document encoding.
Therefore we follow recent work in representation-based Entity Linking
(Gillick et al., 2019) and embed textual knowledge from the clinical domain
into this representation. Our goal is to generalize entity representations, so
the model will be able to align to existing taxonomies without retraining.
Therefore, our entity space must be as complete as possible: it needs to cover
each of the entities that appear in the discourse training data, but also rare
entities that we expect at query time, e.g. in the application. We must
further provide a robust method for predicting unseen entities (Logeswaran et
al., 2019). In contrast to highly specialized entity embeddings constructed
from knowledge graphs or multimodal data (Beam et al., 2018), our generic
approach is based on textual data and allows us to apply the model to
different knowledge bases and domains.

####  3.1.1. Entity Embeddings

Our goal is to create a mapping of each entity in the knowledge base
$E\in\mathcal{K}$ identified by its ID into a low-dimensional entity vector
space $\mathbb{E}\subset\mathbb{R}^{d}$222we use $d$ as a placeholder for all
embedding vector sizes, even if they are not equal. We train an embedding by
minimizing the loss for predicting the entity from sentences
$s\in\mathcal{D}_{E}$ in the entity descriptions:

(1)
$\displaystyle\mathcal{L}_{\text{entity}}(\Theta)=-\text{log}\prod_{E\in\mathcal{K}}\prod_{s\in\mathcal{D}_{E}}p_{\Theta}\bigl{(}E\text{.id}\mid
s\bigr{)}$

where $\Theta$ denotes the parameters required to approximate the probability
$p$. We optimize $\Theta$ using a bidirectional Long Short-Term Memory (BLSTM)
(Hochreiter and Schmidhuber, 1997) to predict the entity ID $E\text{.id}$ from
the words $w_{i}\in s$. We encode $w_{i}$ using Fasttext embeddings
(Bojanowski et al., 2017) and use bloom filters (Serrà and Karatzoglou, 2017)
to compress $E\text{.id}$ into a hashed bit encoding, allowing for less model
parameters and faster training.

(2) $\displaystyle p_{\Theta}\bigl{(}E\text{.id}\mid s\bigr{)}$
$\displaystyle=p_{\Theta}\bigl{(}E\text{.id}\mid w_{1},\ldots,w_{N}\bigr{)}$
$\displaystyle\approx
p\bigl{(}\text{bloom}(E\text{.id})\mid\text{BLSTM}_{\Theta}(w_{1},\ldots,w_{N})\bigr{)}$

Subsequently, we extend the approach of Palangi et al. (2016) and define the
embedding function $\epsilon$ as the average output of the hidden word states
$\vec{g}_{k}$ and $\reflectbox{$\vec{\reflectbox{$g$}}$}_{k}$ at the first
respectively last time step:

(3) $\displaystyle\vec{g}_{k}$
$\displaystyle=\text{LSTM}_{\Theta}(\vec{g}_{k-1},w_{k})$
$\displaystyle\reflectbox{$\vec{\reflectbox{$g$}}$}_{k}$
$\displaystyle=\text{LSTM}_{\Theta}(\reflectbox{$\vec{\reflectbox{$g$}}$}_{k+1},w_{k})$
(4)
$\epsilon(s)=\frac{\vec{g}_{T}+\reflectbox{$\vec{\reflectbox{$g$}}$}_{1}}{2}$

Finally, we generate entity embeddings $\epsilon_{E}\in\mathbb{E}$ by applying
the embedding function to all descriptions available. In case of unseen
entities, the embedding can be generated on-the-fly:

(5)
$\epsilon_{E}=\begin{cases}\frac{1}{|\mathcal{D}_{E}|}\sum_{s\in\mathcal{D}_{E}}\epsilon(s)&\text{
if }E\in\mathcal{K}\\\ \epsilon(E.\text{mention})&\text{ if
}E\notin\mathcal{K}\end{cases}$

####  3.1.2. Training Data

We train the entity representation for diseases, syndromes and health problems
using textual descriptions from various sources:
Wikidata333https://www.wikidata.org/wiki/Q12136,
UMLS444https://uts.nlm.nih.gov, GARD555https://rarediseases.info.nih.gov,
Wikipedia abstracts, and the Diseases
Database666http://www.diseasesdatabase.com. In total, the knowledge base
contains over 27,000 entities identified by their Wikidata ID. We trained
roughly 9,700 common entities with text from Wikipedia abstracts, while we
used for rare entities only their name and short description texts.

###  3.2. Aspect Space

The second part of our problem is to represent the aspect in the query tuple.
Here, we expect a wide range of clinical facets and we do not want to limit
the users of our system to a specific terminology. Instead, we train a low-
dimensional aspect vector space $\mathbb{A}\subset\mathbb{R}^{d}$ using the
Fasttext skip-gram model (Bojanowski et al., 2017) on medical Wikipedia
articles. This approach places words with similar semantics nearby in vector
space and allows queries with morphologic variations using subword
representations.

To find all possible aspects, we adopt prior work (Arnold et al., 2019) and
collect all section headings from the medical Wikipedia articles. These
headings typically consist of 1–3 words and describe the main topic of a
section. We apply moderate preprocessing (lowercase, remove punctuation, split
at “and—&”) to generate aspect embeddings $\alpha_{A}\in\mathbb{A}$ using a
BLSTM encoder $\alpha(s)$ with the same architecture as discussed above:

(6)
$\alpha_{A}=\frac{1}{|\mathcal{D}_{A}|}\sum_{s\in\mathcal{D}_{A}}\alpha(s)$

We train the embedding with over 577K sentences from Wikipedia (see Table 1).
We observe that there is a vocabulary mismatch in the headings so that
potentially synonymous aspects are frequently labeled with different headings,
e.g. “types” / “classification” or “signs” / “symptoms” / “presentation” /
“characteristics”. However, it is also possible that in some contexts these
aspects are hierarchically structured, e.g. “presentation” refers to the
visible forms of a symptom. Our vector-space representation reflects these
similarities, so it is possible to distinguish between these nuances at query
time.

Table 1. Distribution of the top 32 headings (60% of 577K total occurrences) contained in our training set. Numbers are given as cumulative sum. We observe that these headings cover the most important aspects for differential diagnosis. Heading | cusum | Heading | cusum
---|---|---|---
information (abstract) | 9.6% | mechanism | 54.1%
treatment | 16.0% | culture | 54.7%
diagnosis | 22.3% | society | 55.3%
symptoms | 28.2% | research | 55.9%
signs | 33.0% | risk factors | 56.4%
causes | 37.1% | presentation | 56.9%
history | 39.3% | differential diagnosis | 57.3%
pathophysiology | 41.4% | surgery | 57.7%
management | 43.4% | treatments | 58.1%
epidemiology | 45.4% | pathogenesis | 58.4%
cause | 47.3% | medications | 58.7%
classification | 48.9% | complications | 59.0%
prognosis | 50.4% | characteristics | 59.3%
prevention | 51.7% | medication | 59.5%
types | 52.5% | other animals | 59.7%
genetics | 53.3% | pathology | 60.0%

###  3.3. Query Representation

Finally, we represent the query as a tuple of entity and aspect embeddings
using vector concatenation ($\oplus$):

(7) $\displaystyle Q(E,A)$
$\displaystyle=\langle\epsilon_{Q}\in\mathbb{E},\alpha_{Q}\in\mathbb{A}\rangle=\epsilon_{Q}\oplus\alpha_{Q}$

This query encoder constitutes the upper part of our dual encoder architecture
shown in Figure 2. In the next section, we introduce our document
representation that completes the lower part.

##  4\. Contextualized Discourse Vectors

In this section, we introduce Contextual Discourse Vectors (CDV), a
distributed document representation that focuses on coherent encoding of local
discourse in the context of the entire document. The architecture of our model
is shown in Figure 2. We approach the challenges introduced in Section 1 by
reading a document at word and sentence level (Section 4.1) and encoding
sentence-wise representations using recurrent layers at document level
(Section 4.2). We use the representations to measure similarity between every
position in the document and the query (Section 4.3). Our model is trained to
match the entity/aspect vector spaces introduced in Section 3 using self-
supervision (Section 4.4).

[Architecture diagram showing how documents and queries are encoded and
scored.]Architecture diagram flowing from bottom to top. On the bottom, each
word (w) of a sentence in the document is encoded into a sentence embedding
(sigma). These vectors are fed into a BLSTM layer that produces contextualized
discourse vectors (CDV, delta). The CDV vectors are split by a feed-forward
network into entity (epsilon) and aspect (alpha) embeddings. On the top,
entity context is encoded into an entity query embedding (epsilon Q) and an
aspect heading is encoded into an aspect query embedding (alpha Q). In the
center, passages are scored by comparing both sides for each sentence in a
document from left to right.

Figure 2. Neural network architecture for our contextualized document
representation. The contextual discourse vectors (CDV) are generated by a
hierarchical stack of layers: sentence encoder (GloVe/Fasttext/BioBERT) and
document encoder (BLSTM). The query encoder (entity/aspect embeddings) is used
for scoring on sentence level.

###  4.1. Sentence Encoder

The first group of layers in our architecture encodes the plain text of an
entire document into a sequence of vector representations. As we expect long
documents—the average document length in our test sets is over 1,200 words—we
chose to reduce the computational complexity by encoding the document
discourse at sentence-level. It is important to avoid losing document context
and word–discourse interactions (e.g. entity names or certain aspect-specific
terms) during this step. Furthermore, our challenge of _domain adaptability_
requires the sentence encoder to be robust to linguistic and semantic shifts
from text sources that differ from the training data.

Therefore we start at the input layer by encoding all words in a document $D$
into fixed low-dimensional word vectors $w_{1\dots N}\in\mathbb{R}^{d}$ using
pre-trained word embeddings with subword information (see below). Next, we
encode all sentences $s_{1\dots T}\in D$ into sentence representations
$\sigma_{t}\in\mathbb{R}^{d}$ based on the words $w_{k}\in s_{t}$ in the
sentence. This will reduce the number of computational time steps from $N$
words in a document to $T$ sentences. We compare two approaches for this
sentence encoding step:

####  4.1.1. Compositional Sentence Embeddings

As the simplest approach we use an average vector composition of the word
embeddings $w_{k}$ from GloVe (Pennington et al., 2014) or Fasttext
(Bojanowski et al., 2017), which is more robust against out-of-vocabulary
errors:

(8) $\sigma_{\text{avg}}(s)=\frac{1}{\text{len}(s)}\sum_{w_{k}\in s}w_{k}$

####  4.1.2. Pooling-based Sentence Embeddings

Since we want the model to be able to focus on individual words, we apply a
language model encoder. We use the recent BioBERT (Lee et al., 2019), a
transformer model which is pre-trained with a large amount of biomedical
context on sub-word level. To generate sentence vectors from the input
sequence, we use pooling of the attention layers per sentence:

(9)
$\sigma_{\text{pool}}(s)=\text{BioBERT}(w_{s\text{.begin}},\dots,w_{s\text{.end}})$

Finally, we concatenate a positional encoding to the sentence embeddings,
which encodes some rule-based structural flags such as begin/end-of-document,
begin/end-of-paragraph, is-list-item. This encoding helps to guide the
document encoder through the structure of a document.

Data flow diagram that visualizes the three stages training, indexing and ad-
hoc retrieval.

Figure 3. The entire answer passage retrieval process with three stages. (1)
We train the discourse-aware document representation model using self-
supervision on Wikipedia articles. (2) At indexing time, the model is applied
once to the entire test corpus of unseen healthcare articles. The discourse
vectors are saved into a vector index. (3) A query is retrieved by ranking
similarity scores between the query representation and all sentence-level
discourse vectors in the candidate passages.

###  4.2. Document Encoder

The second group of layers in our architecture encodes the sequence of
sentences over the document. The objective of these layers is to transform the
word/sentence input space into discourse vector space—which will later match
with query entity and aspect spaces—in the context of the document. To achieve
_contextual coherence_ , we use the entire document as input for a recurrent
neural network with parameters $\Theta$, which we optimize at training time to
minimize the loss over the sequence:

(10)
$\mathcal{L}_{\text{doc}}(\Theta)=-\text{log}\prod_{t=1}^{T}p_{\Theta}\bigl{(}\epsilon(s_{t}),\alpha(s_{t})\mid\sigma(s_{1}),\dots,\sigma(s_{T})\bigr{)}$

We adopt the architecture of SECTOR (Arnold et al., 2019) and use
bidirectional LSTMs to read the document sentence-by-sentence. We use a final
dense layer (matrix $W_{he}$ and bias $b_{e}$) to produce the local discourse
vectors $\delta_{1\dots T}$ for every sentence in $D$.

(11) $\displaystyle\vec{h}_{t}$
$\displaystyle=\text{LSTM}_{\Theta}\bigl{(}\vec{h}_{t-1},\sigma(s_{t})\bigr{)}$
$\displaystyle\reflectbox{$\vec{\reflectbox{$h$}}$}_{t}$
$\displaystyle=\text{LSTM}_{\Theta}\bigl{(}\reflectbox{$\vec{\reflectbox{$h$}}$}_{t+1},\sigma(s_{t})\bigr{)}$
$\displaystyle\delta_{t}$
$\displaystyle=\text{tanh}\bigl{(}W_{he}(\vec{h}_{t}\oplus\reflectbox{$\vec{\reflectbox{$h$}}$}_{t})+b_{e}\bigr{)}$

The CDV matrix $C_{D}=[\delta_{1},\dots,\delta_{T}]$ is our discourse-aware
document representation which embeds all information necessary to decode
contextualized entity and aspect information for $D$.

###  4.3. Passage Scoring

The center layer in our architecture addresses our main task objective: find
the passages with highest similarity to the query. In Section 3, we described
generalized vector spaces for entities $\epsilon\in\mathbb{E}$ and aspects
$\alpha\in\mathbb{A}$ that we use as query representation $Q$ for high _task
coverage_. We train our discourse vectors $C_{D}$ to share the same vector
spaces $\mathbb{E}$ and $\mathbb{A}$. This enables us to run _efficient neural
information retrieval_ of multiple ad-hoc queries $Q$ over the pre-computed
CDV vectors $C_{D}$ later without having to re-run inference on the document
encoder for each query. We store all vectors $\delta_{t}$ in an in-memory
vector index that allows us to efficiently retrieve approximate nearest
neighbors using cosine distance. Figure 3 shows the overall process of
training, indexing and ad-hoc answer retrieval. Because we reuse entity and
aspect embeddings for training, our document model ‘inherits’ the properties
from these spaces, e.g. robustness for unseen and rare entities or aspects.

####  4.3.1. Discourse Decoder

To decode the individual entity and aspect predictions
$\hat{\epsilon}_{t},\hat{\alpha}_{t}$ from $\delta_{t}\in C_{D}$, we utilize
two learned decoder matrices $W_{\delta\epsilon},W_{\delta\alpha}$ with bias
terms $b_{\epsilon},b_{\alpha}$. We optimize these parameters by using a
multi-task objective with shared weights (Caruana, 1997) to minimize the
distance to the training labels $\epsilon_{t},\alpha_{t}$:

(12) $\displaystyle\hat{\epsilon}_{t}$
$\displaystyle=\text{tanh}(W_{\delta\epsilon}\delta_{t}+b_{\epsilon})$
$\displaystyle\hat{\alpha}_{t}$
$\displaystyle=\text{tanh}(W_{\delta\alpha}\delta_{t}+b_{\alpha})$
$\displaystyle\mathcal{L}_{\text{cdv}}(\Theta)$
$\displaystyle=\frac{1}{T}\sum_{t=1}^{T}\bigl{(}\lVert\hat{\epsilon}_{t}-\epsilon_{t}\rVert+\lVert\hat{\alpha}_{t}-\alpha_{t}\rVert\bigr{)}$

####  4.3.2. Sentence Scoring

To compute similarity scores at query time, we pick up our query
representation (Eq. 7) and compute the semantic similarity between $Q$ and
each contextual discourse vector $\delta_{t}$ in the vector index. To achieve
low latency, we use cosine similarity between the decoded entity and aspect
representations:

(13) $\displaystyle\text{score}_{t}\bigl{(}Q(E,A),C_{D}\bigr{)}$
$\displaystyle=\text{cosine}\bigl{(}\epsilon_{Q}\oplus\alpha_{Q},\hat{\epsilon}_{t}\oplus\hat{\alpha}_{t}\bigr{)}$
$\displaystyle=\frac{(\epsilon_{Q}\oplus\alpha_{Q})^{\text{T}}(\hat{\epsilon}_{t}\oplus\hat{\alpha}_{t})}{\lVert\epsilon_{Q}\oplus\alpha_{Q}\rVert\lVert\hat{\epsilon}_{t}\oplus\hat{\alpha}_{t}\rVert}$

####  4.3.3. Answer Retrieval

The scoring operation yields a sentence-level histogram $\text{score}_{1\dots
T}(Q,C_{D})\in[0,1]$ which describes the similarity between query and every
sentence in a document. At this point, we have the opportunity to select a
coherent set of sentences as answers similar to (Arnold et al., 2019).
However, because all healthcare datasets that we use for evaluation in this
paper already provide passage boundaries, we leave this step for future work.
Instead, we use the average sentence score per passage for answer retrieval:

(14) $\displaystyle\text{score}(Q,P)$
$\displaystyle=\frac{1}{\text{len}(P)}\sum_{s_{t}\in
P}\text{score}_{t}(Q,C_{D})$

Figure 4 shows the scoring curves divided into entity $Q(E)$, aspect $Q(A)$
and average $\text{score}\bigl{(}Q(E,A),P\bigr{)}$. It is clearly visible that
the model coherently predicts long-range dependencies for the entity “IgA
nephropathy” over the entire document. The aspect similarity with “symptoms”
is much more focused on single sentences.

[Three plots that show the similarity of entity, aspect and passage with the
query.]Three plots that show the similarity of entity, aspect and passage
(combined) of each sentence with the query. The area with highest similarity
is highlighted.

Figure 4. CDV model predictions for query “symptoms of IgA nephropathy” on the
example document. The histogram shows the similarity score of the discourse
vector with the query over sentences $t=1\dots T$ from left to right.

###  4.4. Self-supervised Training

We train a generalized CDV model for all evaluation tasks by jointly
optimizing all model parameters from sentence encoder, document encoder and
passage scoring layers on a training set. For this task, we use the textual
data about diseases and health problems available from Wikipedia. This process
is self-supervised, because there exist no labeled query-answer pairs for
these documents. Instead, we assign for each sentence $s_{t}\in D$ a set of
related entities $E$ and aspects $A$ using simple heuristics:

(15) $\displaystyle E(s_{t},D)$
$\displaystyle=\\{E\mid\text{title}(D)=E\vee\text{contains\\_link}(s_{t},E)\\}$
$\displaystyle A(s_{t},D)$ $\displaystyle=\\{A\mid\text{heading}(s_{t})=A\\}$

We collected over 8,600 articles for training and removed all instances
contained in any of the test sets. The collection contains information about
over 8K entities and 15K aspects (see Table 2).

####  4.4.1. Discourse Encoding

We create the target objectives for training using the average of the label
embeddings contained in the training entities $E(s_{t},D)$ and aspects
$A(s_{t},D)$ on sentence level:

(16) $\displaystyle\begin{aligned} \epsilon_{t}&=\frac{\sum_{E\in
E(s_{t},D)}\epsilon_{E}}{|E(s_{t},D)|}\\\ \alpha_{t}&=\frac{\sum_{A\in
A(s_{t},D)}\alpha_{A}}{|A(s_{t},D)|}\end{aligned}$

####  4.4.2. Optimized Loss Function

We observe a strong imbalance of entity and aspect labels over the course of a
single document, for example when passages contain lists (very short
sentences), rare entities or have uncommon headlines. To give the network the
ability to capture these anomalies, especially with larger batch sizes, we use
a robust loss function (Barron, 2019) which resembles a smoothed form of Huber
Loss (Huber, 1992):

(17)
$\displaystyle\mathcal{L}_{\text{cdv+}}(\Theta)=\frac{1}{T}\sum_{t=1}^{T}\left(\sqrt{1+\left(\frac{\lVert\hat{\epsilon}_{t}-\epsilon_{t}\rVert+\lVert\hat{\alpha}_{t}-\alpha_{t}\rVert}{4}\right)^{2}}-1\right)$

In the next section, we apply our CDV model to a healthcare answer retrieval
task.

##  5\. Experimental Results

We evaluate our CDV model and 14 baseline methods in an answer passage
retrieval task. All models are trained using self-supervision on Wikipedia
texts and applied as zero-shot task (Palatucci et al., 2009) (i.e. without
further fine-tuning) to three diverse English healthcare datasets
_WikiSection_ , _MedQuAD_ and _HealthQA_.

###  5.1. Evaluation Setup

As queries, we use tuples of the form
$\langle\text{entity},\text{aspect}\rangle$. Because our task requires to
retrieve the answers from over 4,000 passages and the interaction-based models
in our comparison require computationally expensive pairwise inference, we
evaluate all numbers on a re-ranking task (Gillick et al., 2018). We follow
the setup of Logeswaran et al. (2019) and use BM25 (Robertson et al., 1995) to
provide each model with a pre-filtered set of 64 potentially relevant passage
candidates777This choice covers 80-91% of all true answers (depending on the
dataset) as trade-off between task complexity and real-world applicability.
The numbers reported for HealthQA in the original paper were evaluated by re-
ranking ten candidates (one relevant, 3 partially relevant and 6 irrelevant)
and are therefore not comparable.. To facilitate full recall in this model
comparison, we add missing true answers to the candidates if necessary by
overwriting the lowest-ranked false answers in the list and shuffle
afterwards. We rank the candidate answers using exhaustive nearest neighbor
search and leave the evaluation of indexing efficiency for future work. Next,
we describe the datasets, metrics and methods used in our experiments.

Table 2. Statistics of our training and evaluation data sets. Dataset | Wikipedia | $\Rightarrow$ | WS | MQ | HQ
---|---|---|---|---|---
Split | train |  | test | test | test
# documents | 8,605 |  | 716 | 1,111 | 178
# passages | 53,477 |  | 4,373 | 3,762 | 1,109
# entities | 8,605 |  | 716 | 1,100 | 221
# aspects | 15,028 |  | 27 | 15 | 21
# queries | N/A |  | 4,178 | 3,294 | 1,045
avg words/doc | 977.6 |  | 1,396.7 | 811.1 | 1,449.4
avg sents/doc | 43.5 |  | 63.7 | 48.0 | 82.5
avg passages/doc | 6.2 |  | 6.1 | 3.4 | 6.2
avg words/passage | 221.8 |  | 228.6 | 237.9 | 232.6
avg sents/passage | 9.8 |  | 10.4 | 14.1 | 13.2
avg words/sent | 22.7 |  | 21.9 | 16.9 | 17.6
Table 3. Results for the answer passage retrieval task on three healthcare datasets. All models were trained using the self-supervised Wikipedia training set and applied without fine-tuning. Queries were evaluated by ranking 64 given candidates from the respective test sets. As queries we used $\langle\text{entity},\text{aspect}\rangle$ tuples in a representation suitable for the individual model. Model | WikiSectionQA | MedQuAD | HealthQA
---|---|---|---
all trained on Wikipedia | $R@1$ | $R@10$ | MAP | $R@1$ | $R@10$ | MAP | $R@1$ | $R@10$ | MAP
_Term-based models_ |  |  |  |  |  |  |  |  |
TF-IDF (Jones, 1972) | 17.10 | 64.99 | 31.77 | 23.83 | 82.84 | 42.66 | 17.46 | 71.54 | 34.47
BM25 (Robertson et al., 1995) | 23.87 | 71.26 | 38.89 | 29.48 | 86.11 | 48.89 | 22.55 | 73.27 | 38.45
_Representation-based models_ |  |  |  |  |  |  |  |  |
ARC-I (Hu et al., 2014) | 1.61 | 13.69 | 6.90 | 1.98 | 19.22 | 8.47 | 1.38 | 13.87 | 6.87
DSSM (Huang et al., 2013) | 22.82 | 74.31 | 39.02 | 13.11 | 55.92 | 27.38 | 10.50 | 46.44 | 22.04
C-DSSM (Shen et al., 2014) | 9.59 | 53.12 | 22.82 | 9.67 | 47.54 | 22.12 | 10.56 | 58.30 | 25.37
_Interaction-based models_ |  |  |  |  |  |  |  |  |
ARC-II (Hu et al., 2014) | 10.38 | 53.62 | 23.61 | 9.19 | 47.58 | 21.66 | 11.26 | 58.85 | 26.09
DRMM (Guo et al., 2016) | 24.96 | 67.56 | 39.24 | 34.52 | 82.35 | 51.51 | 21.80 | 80.24 | 40.03
MatchPyramid (Pang et al., 2016) | 18.53 | 64.21 | 33.12 | 25.14 | 72.33 | 41.37 | 19.24 | 73.79 | 37.22
aNMM (Yang et al., 2016a) | 4.77 | 32.17 | 14.03 | 7.15 | 37.18 | 17.08 | 3.74 | 27.20 | 12.07
KNRM (Xiong et al., 2017) | 16.96 | 61.03 | 31.04 | 16.86 | 61.35 | 31.35 | 22.94 | 67.92 | 37.65
CONV-KNRM (Dai et al., 2018) | 34.36 | 77.25 | 48.72 | 42.70 | 84.54 | 57.57 | 33.13 | 85.41 | 50.55
HAR (Zhu et al., 2019) | 45.31 | 84.15 | 58.38 | 55.65 | 93.17 | 69.10 | 43.20 | 88.34 | 58.80
_Combined models_ |  |  |  |  |  |  |  |  |
Duet (Mitra et al., 2017) | 18.34 | 59.13 | 31.74 | 20.50 | 65.91 | 35.28 | 17.27 | 64.81 | 32.13
MVLSTM (Wan et al., 2016) | 30.74 | 76.10 | 45.58 | 36.86 | 86.29 | 53.18 | 26.78 | 84.42 | 45.37
_Our model_ |  |  |  |  |  |  |  |  |
CDV+avg-glove | 59.60 | 95.67 | 72.72 | 34.00 | 80.87 | 50.45 | 37.17 | 84.47 | 53.30
CDV+avg-fasttext | 60.34 | 97.49 | 74.01 | 45.26 | 92.29 | 62.56 | 40.08 | 89.80 | 58.35
CDV+pool-biobert | 65.21 | 97.84 | 77.60 | 39.96 | 91.32 | 58.91 | 43.60 | 88.12 | 59.40

####  5.1.1. Evaluation Datasets

We conduct experiments on three English datasets from the clinical and
healthcare domain (see Table 2). From the documents provided, we use the plain
text of the entire document body during model inference and the segmentation
information for generating the passage candidates. From all queries provided,
we use the entity labels (mention text, Wikidata ID) and aspect labels (UMLS
canonical name). If entity and aspect identifiers were not provided by the
dataset, we added them manually by asking three annotators from clinical
healthcare to label them.

_WikiSectionQA_ (Arnold et al., 2019) (WS) is a large subset of full-text
Wikipedia articles about diseases, labeled with entity identifiers, section
headlines and 27 normalized aspect classes. We extended this dataset for
answer retrieval by constructing query tuples from every section containing
the given entity ID and normalized aspect label. We included abstracts as
“information”, but skipped sections labeled as “other”. We use the en_disease-
test split for evaluation and made sure that none of the documents are
contained in our training data.

_MedQuAD_ (Abacha and Demner-Fushman, 2019) (MQ) is a collection of medical
question-answer pairs from multiple trusted sources of the National Institutes
of Health (NIH): National Cancer Institute (NCI)888https://www.cancer.gov,
Genetic and Rare Diseases (GARD) 999https://rarediseases.info.nih.gov,
Genetics Home Reference (GHR)101010https://ghr.nlm.nih.gov, National Institute
of Diabetes and Digestive and Kidney Diseases (NIDDK)
111111http://www.niddk.nih.gov/health-information/health-topics/, National
Institute of Neurological Disorders and Stroke (NINDS)
121212http://www.ninds.nih.gov/disorders/, NIH Senior Health
131313http://nihseniorhealth.gov/ and National Heart, Lung and Blood Institute
(NHLBI)141414http://www.nhlbi.nih.gov/health/. We left out documents from
Medline Plus due to property rights. Questions are annotated with structured
identifiers for entities (UMLS CUI), aspect (semantic question type) and
contain a long passage as answer. To make this dataset applicable to our
method, we reconstructed the entire documents from the answer passages and
kept only questions about diseases for evaluation. We filtered out documents
with only one passage (these were always labeled “information”) and separated
a random 25% test split from the remaining documents.

_HealthQA_ (Zhu et al., 2019) (HQ) is a collection of consumer health
question-answer pairs crawled from the website
Patient151515https://patient.info. The answer passages were generated from
sections in the documents and annotated by human labelers with natural
language questions. We reconstructed the full documents from these sections.
Additionally, our annotators added structured entity and aspect labels to all
questions in the test split. Although some questions are not about diseases,
we kept all of them to remain comparable with related work.

####  5.1.2. Evaluation Metrics

For all ranking experiments, we use _Recall at top K_ (R@K) and _Mean Average
Precision_ (MAP) metrics. While R@1 measures if the top-1 answer is correct or
not (similar to a question answering task), we also report R@10, which
corresponds with the ability to retrieve all correct answers in a top-10
results list, and MAP, which considers the entire result list.

####  5.1.3. Baseline Methods

We evaluate two term-based matching functions as baseline: TF-IDF (Jones,
1972) and BM25 (Robertson et al., 1995). We used the implementation in Apache
Lucene 8.2.0161616https://lucene.apache.org to retrieve passages containing
entity and aspect of a query, e.g. “IgA nephropathy symptoms” from the index
of all passages in the test dataset.

Additionally, we evaluate the following document matching methods from the
literature: ARC-I and ARC-II (Hu et al., 2014), DSSM (Huang et al., 2013),
C-DSSM (Shen et al., 2014), DRMM (Guo et al., 2016), MatchPyramid (Pang et
al., 2016), aNMM (Yang et al., 2016a), Duet (Mitra et al., 2017), MVLSTM (Wan
et al., 2016), KNRM (Xiong et al., 2017), CONV-KNRM (Dai et al., 2018) and HAR
(Zhu et al., 2019). For implementing these models, we followed Zhu et al.
(2019) and used the open source implementation MatchZoo (Guo et al., 2019)
with pre-trained glove.840B.300d vectors (Pennington et al., 2014). All models
were trained with our self-supervised Wikipedia training set using queries
containing the entity and lowercase heading, e.g. “IgA nephropathy ; symptoms”
and applied to the test sets using queries of the same structure, instead of
natural language questions.

###  5.2. Implementation Details

We implement our models with the following configurations: For the sentence
encoding, we use either glove.6B.300d pre-trained GloVe vectors (+avg-glove),
128d fine-tuned Fasttext embeddings (+avg-fasttext) or the 768d pre-trained
BioBERT (Lee et al., 2019) language model (+pool-biobert). For the document
encoding, we use two LSTM layers (one forward, one backward) with 512
dimensions each, a discourse vector dense layer with 256 dimensions, L2 batch
normalization and tanh activation. The discourse decoder is a 128-dimensional
output layer with tanh activation and Huber loss. The network is trained with
stochastic gradient descent over 50 epochs using the ADAM optimizer with a
batch size of 16 documents, a learning rate of $10^{-3}$, $0.975$ exponential
decay per epoch, $0.0$ dropout and $10^{-4}$ weight decay regularization. We
chose these parameters using hyperparameter search on the WikiSection
validation set. During training, we restrict the maximum document length to
396 sentences and maximum sentence length to 96 tokens, due to memory
constraints on the GPU.

The entity and aspect embeddings are trained with 128d Fasttext embeddings,
followed by BLSTM and dense embedding layers, each with 128 dimensions and
tanh activations. The output layer is configured with 1024 dimensions, sigmoid
activation and BPMLL loss (Zhang and Zhou, 2006) to predict the Bloom filter
hash ($k=5$) of the entity or aspect. The network is trained similarly to the
CDV model, but we use 5 epochs with a batch size of 128 sentences, a learning
rate of $10^{-3}$ and $0.5$ dropout.

###  5.3. Results

Table 3 shows the results on the answer passage retrieval task using CDV and
document matching models on three healthcare datasets. We observe that CDV
consistently achieves significantly better results than all term-based,
representation-based and combined models across all datasets. In comparison
with pairwise interaction-based models, our representation-based retrieval
model outperforms all tested models on average, scores best on WikiSection and
HealthQA and second best on the MedQuAD dataset. Retrieval time per query is
247ms ($\pm$43ms) on average. Figure 5 further shows that we correctly match
between 67.5% and 91.4% of entities in the datasets and resolve 49.4% to 66.3%
of all aspects.

##### Comparison of Model Architectures

Our query model is able to match most of the questions in the entity/aspect
scheme (see Section 5.4 for exceptions). The results show that term-based TF-
IDF and BM25 models can solve the healthcare retrieval task sufficiently with
R@10 $>70\%$. In contrast, none of the representation-based re-ranking models
can achieve similar performance, except DSSM on WikiSectionQA. Most of the
recent interaction-based and combined models outperform BM25 and have
significant advantages on the MedQuAD dataset, which contains a large amount
of generated information that can be matched exactly. We follow that simple
word-level interactions are important for this task and representation-based
models trade off this property for fast retrieval times.

##### Background Knowledge

Our CDV model performs well on all data sets, but shows a significant
advantage on the Wikipedia-based WikiSectionQA dataset. Although all models
are trained on the same data, the only model with similar behavior is DSSM.
One possible reason is that entity embeddings are an important source for
background information, and these are mainly based on Wikipedia descriptions.
99.9% of the WikiSectionQA entities are covered in our embedding, 97.1% on
MedQuAD and only 69.29% on HealthQA, because it does not only contain
diseases. Additionally, sentence embeddings provide different levels of
background knowledge and language understanding. The pre-trained GloVe
embedding can handle the task well, but is outperformed by our fine-tuned
Fasttext embedding and the large BioBERT language model.

##### Domain Adaptability and Task Coverage

Figure 6 shows the performance of our CDV+avg-fasttext model across all data
sources, most of them contained in MedQuAD. This distribution reveals that our
model top-1 accuracy is stable in the adaptation to most sources except
National Cancer Institute (CancerGov). However, we notice that R@10
performance is high among all sources except SeniorHealth. Figure 7 shows that
R@10 performance across the most frequent aspects is over 93% in most cases,
but with varying top-1 recall. We will address these errors in Section 5.4.

##### Impact of Contextual Dependencies

An important feature of our CDV model is that all score predictions are
calculated on sentence level with respect to long-range context across the
entire document. In Figure 4, we observe that the model is able to predict the
entity (top curve) consistently over the document, although there are many
coreferences in Wikipedia text. The aspect curve (center) clearly shows the
beginning of the expected section “Symptoms” and the model is uncertain for
the following sentences. Finally, the average score (bottom curve) shows a
coherent prediction.

[Plot showing the proportions for match, entity match, aspect match and
mismatch.]Plot showing the proportions for match (42–62%), entity match
(20–34%), aspect match (4–17%) and mismatch (4–17%) for three datasets.

Figure 5. Entity/aspect matching (values in %) observed on all examples in the
three evaluation datasets.

[Bar plot showing the individual Recall values for each data source.]Bar plot
showing the individual Recall@1/10 values for data sources Senior Health
(61/75%), Wikipedia (60/98%), NHBLI (52/92%), GARD (50/88%), NINDS (46/95%),
NIDDK (43/81%), GHR (43/98%), Patient (40/90%) and CancerGov (33/91%).

Figure 6. R@1 and R@10 performance of the CDV-EA+pool-biobert model across all
data sources.

[Bar plot showing the individual Recall values for each aspect.]Bar plot
showing the individual Recall@1 values aspects information (47%), symptoms
(53%), treatment (69%), diagnosis (77%), causes (58%), inheritance (50%),
prognosis (38%), prevalence (51%), mutations (34%), pathogenesis (62%),
epidemiology (90%), management (43%), classification (39%), research (45%),
prevention (37%), history (7%) and risk factors (36%). Recall@10 varies
between 89–100% except for history with 73%.

Figure 7. R@1 and R@10 prediction performance on the 17 most frequent aspects
in all test sets (of 34 total).

###  5.4. Discussion and Insights

[Plot showing the proportion of entity and aspect errors among the mismatched
queries.]Plot showing the proportion of entity errors (related 12–26%, similar
name 1-24%, not a disease 0–22%, other mismatch) and aspect errors (related
4–34%, other mismatch) among the mismatched queries the three datasets.

Figure 8. Error classes for entity and aspect mismatch (values in %) from
manual analysis of 150 mismatched queries.

We perform an error analysis on the predictions of the CDV+avg-fasttext model
to identify main reasons for answer misranking. For this purpose we analyse
samples in which the model ranks a wrong passage at the top-1 position. We
look at 50 random mismatched samples per dataset to understand the individual
challenges per source. We discuss the main findings in the following.

##### Related Entities

Figure 8 shows that a main source of entity errors comes from selecting
passages that belong to related entities. This includes entities that are
superclasses or subclasses of the gold truth, e.g. selecting a passage
covering “Diarrhea” when “Chronic Diarrhea in Children” is the query entity.
These errors are most significant in WikiSectionQA and MedQuAD, because
HealthQA covers mostly common diseases. We especially observe this in samples
from Genetic Home Reference and National Cancer Institute. Figure 6 shows that
R@1 is low for samples from these sources, whereas their R@10 is high. That is
because genetic conditions and cancer types inherently contain entities with
very similar names and descriptions. For instance, we see “Spastic Paraplegia
Type 8” falsely resolved to “Spastic Paraplegia Type 11”. As the
representations are close to each other in vector space, the correct samples
are almost always found within the top-10 ranked candidates, corresponding
with the high R@10.

##### Related Aspects

Likewise, we observe that in HealthQA 34% and in WikiSectionQA 16% of aspects
are mismatched to related aspects. Figure 7 shows the distribution of aspects
and the model’s ability to resolve them. It is salient that some aspects are
especially difficult to resolve. Aside from the fact that these aspects are in
the long tail, a further analysis reveals that they are often resolved to
related aspects. For example, passages covering “classification” are often
very similar and therefore confused with passages about “diagnosis” and
“symptoms”. The same holds true for “prognosis” and “management”. Queries
asking for “prevalence” of a disease are often resolved to “information”
passages, because disease frequency is often mentioned in these introductory
texts. In general, passages about related aspects often share similar tokens
and document context, which makes their distinction more difficult.

##### Out-of-Scope Questions.

25% of queries from HealthQA contain entities that are no diseases but
procedures, drugs or other entity types. As our model is trained on textual
data covering diseases only, we do not expect it to fully resolve these
entities. However, we observe that the model is capable of finding the correct
passage for 23% of unseen entities. This shows that while our model is not
trained on such entity types, the fallback embedding described in 3.1.1 still
allows to generalize even to non-diseases in these cases.

##### Evaluation vs. Real-World Application

We further identified a number of errors related to the structure of the
evaluation, that would be less problematic or even beneficial in real-world
application. The model frequently ranks passages to the top which answer the
query but have a differing aspect assigned. We observe this in 24% of analysed
samples from WikiSectionQA and 18% from HealthQA. This often seems to be
caused by the non-discrete nature of topical aspects. In practice a passage
can cover more than one aspect, but our evaluation does currently not capture
this ambiguity. Additionally we find some mismatches between passages and
their ground-truth aspects, which can be ascribed to writing errors in
WikiSectionQA and labeling errors in HealthQA. Aspects in MedQuAD are less
ambiguous in general and only 4% fall into this error class. Figure 5 shows
that the model therefore resolves more aspects correctly for MedQuAD queries.

##### Irrelevant Text within Passage Boundaries

Another finding is that 28% of analysed samples from the MedQuAD dataset
contain boilerplate text unrelated to a specific entity. The boilerplate
includes repeated text such as information about how data was collected. In
this case our model is able to detect relevant parts of a passage (see Figure
4), but the remaining irrelevant sentences lead to a worse ranking of the
passage. Evaluating with flexible passage boundaries would eliminate this
issue and be a better match for real-world scenarios, in which the interest of
a medical professional is mainly focused on non-boilerplate parts of a
document.

##### Complex Questions

We find that most questions in our evaluation can be represented as tuples of
entity and aspect without information loss. However, in 4% of analysed queries
in the HealthQA dataset we see a mismatch between question and query. For
instance, the question “How common is OCD in Children and Young People?” which
is more specific than the assigned query tuple “Obsessive-compulsive disorder”
and “prevalence”. Different solutions are possible for representing more
complex queries, e.g. by composing multiple queries during retrieval. We leave
these questions for future research.

##  6\. Conclusions and Future Work

We present CDV, a contextualized document representation that uses structured
entity/aspect queries to retrieve answer passages from long healthcare
documents. Our model is based on a hierarchical dual encoder architecture
which combines interaction-based feature encoding with low-latency continuous
retrieval. In comparison to previous approaches, CDV is able to integrate
document context into its representations, which helps to resolve long-range
dependencies normally not visible to passage re-ranking models. We train a
self-supervised model on medical Wikipedia texts and show that it applies to
three healthcare answer retrieval tasks best or second best, compared to 14
strong baseline models. We trained all models using the same data and provide
structured labels on three existing datasets for the evaluation of this task.

In future work, we will address the transfer of the CDV model to different
languages, rare diseases and more complex question types. Another open
challenge is the extraction of passage boundaries during retrieval.
Furthermore, it is interesting to see how fine-tuning the model on supervised
data will improve retrieval performance.

###### Acknowledgements.

Our work is funded by the German Federal Ministry of Economic Affairs and
Energy (BMWi) under grant agreement 01MD19003b (PLASS) and 01MD19013D (Smart-
MD).

## References

  * (1)
  * Abacha and Demner-Fushman (2019) Asma Ben Abacha and Dina Demner-Fushman. 2019.  A Question-Entailment Approach to Question Answering.  _BMC Bioinformatics_ 20, 1 (2019), 511.
  * Abacha et al. (2019) Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019\.  Overview of the Mediqa 2019 Shared Task on Textual Inference, Question Entailment and Question Answering. In _Proceedings of the 18th BioNLP Workshop and Shared Task_. 370–379.
  * Adolphs et al. (2011) Peter Adolphs, Martin Theobald, Ulrich Schafer, Hans Uszkoreit, and Gerhard Weikum. 2011\.  YAGO-QA: Answering Questions by Structured Knowledge Queries. In _Fifth International Conference on Semantic Computing_. IEEE, 158–161.
  * Arnold et al. (2019) Sebastian Arnold, Rudolf Schneider, Philippe Cudré-Mauroux, Felix A. Gers, and Alexander Löser. 2019.  SECTOR: A Neural Model for Coherent Topic Segmentation and Classification.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 169–184.
  * Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016.  A Latent Variable Model Approach to Pmi-Based Word Embeddings.  _Transactions of the Association for Computational Linguistics_ 4 (2016), 385–399.
  * Barron (2019) Jonathan T Barron. 2019\.  A General and Adaptive Robust Loss Function. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4331–4339.
  * Beam et al. (2018) Andrew L Beam, Benjamin Kompa, Allen Schmaltz, Inbar Fried, Griffin Weber, Nathan Palmer, Xu Shi, Tianxi Cai, and Isaac S Kohane. 2018.  Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data. In _Pacific Symposium on Biocomputing_ , Vol. 25. 295–306.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics_ 5 (2017), 135–146.
  * Caruana (1997) Rich Caruana. 1997\.  Multitask Learning.  _Machine learning_ 28, 1 (1997), 41–75.
  * Cheng (2004) Grace YT Cheng. 2004\.  A Study of Clinical Questions Posed by Hospital Clinicians.  _Journal of the Medical Library Association_ 92, 4 (2004), 445\.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 615–621.
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018\.  Convolutional Neural Networks for Soft-Matching n-Grams in Ad-Hoc Search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. ACM, 126–134.
  * Fujiwara et al. (2018) Toyofumi Fujiwara, Yasunori Yamamoto, Jin-Dong Kim, Orion Buske, and Toshihisa Takagi. 2018\.  PubCaseFinder: A Case-Report-Based, Phenotype-Driven Differential-Diagnosis System for Rare Diseases.  _The American Journal of Human Genetics_ 103, 3 (2018), 389–399.
  * Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019.  Learning Dense Representations for Entity Retrieval. In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_. ACL, 528–537.
  * Gillick et al. (2018) Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018\.  End-to-End Retrieval in Continuous Space.  _arXiv:1811.08008 [cs.IR]_ (2018).
  * Gorman et al. (1994) Paul N Gorman, Joan Ash, and Leslie Wykoff. 1994.  Can Primary Care Physicians’ Questions Be Answered Using the Medical Journal Literature?  _Bulletin of the Medical Library Association_ 82, 2 (1994), 140\.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016\.  A Deep Relevance Matching Model for Ad-Hoc Retrieval. In _Proceedings of the 25th ACM International Conference on Information and Knowledge Management_. ACM, 55–64.
  * Guo et al. (2019) Jiafeng Guo, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2019\.  MatchZoo: A Learning, Practicing, and Developing System for Neural Text Matching. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 1297–1300.
  * Hanauer et al. (2015) David A Hanauer, Qiaozhu Mei, James Law, Ritu Khanna, and Kai Zheng. 2015.  Supporting Information Retrieval from Electronic Health Records: A Report of University of Michigan’s Nine-Year Experience in Developing and Using the Electronic Medical Record Search Engine (EMERSE).  _Journal of Biomedical Informatics_ 55 (2015), 290–300.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014\.  Convolutional Neural Network Architectures for Matching Natural Language Sentences. In _Advances in Neural Information Processing Systems_. 2042–2050.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning Deep Structured Semantic Models for Web Search Using Clickthrough Data. In _Proceedings of the 22nd ACM International Conference on Information & Knowledge Management_. ACM, 2333–2338.
  * Huang et al. (2006) Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006\.  Evaluation of PICO as a Knowledge Representation for Clinical Questions. In _AMIA Annual Symposium Proceedings_ , Vol. 2006. AMIA, 359.
  * Huber (1992) Peter J. Huber. 1992\.  Robust Estimation of a Location Parameter.  In _Breakthroughs in Statistics_. Springer, 492–518.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  PubMedQA: A Dataset for Biomedical Research Question Answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_. 2567–2577.
  * Jones (1972) Sparck K. Jones. 1972\.  A Statistical Interpretation of Term Specificity and Its Application to Retrieval.  _Journal of Documentation_ 28, 1 (1972), 11–21.
  * Keikha et al. (2014) Mostafa Keikha, Jae Hyun Park, W. Bruce Croft, and Mark Sanderson. 2014.  Retrieving Passages and Finding Answers. In _Proceedings of the 2014 Australasian Document Computing Symposium_. ACM, 81.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: A Pre-Trained Biomedical Language Representation Model for Biomedical Text Mining.  _Bioinformatics_ (2019), 1–7.
  * Logeswaran et al. (2019) Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019.  Zero-Shot Entity Linking by Reading Entity Descriptions. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 3449–3460.
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017\.  Learning to Match Using Local and Distributed Representations of Text for Web Search. In _Proceedings of the 26th International Conference on World Wide Web_. IW3C2, 1291–1299.
  * Nanni et al. (2017) Federico Nanni, Bhaskar Mitra, Matt Magnusson, and Laura Dietz. 2017.  Benchmark for Complex Answer Retrieval. In _Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval_. ACM, 293–296.
  * Palangi et al. (2016) Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016\.  Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval.  _IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)_ 24, 4 (2016), 694–707.
  * Palatucci et al. (2009) Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. 2009.  Zero-Shot Learning with Semantic Output Codes. In _Advances in Neural Information Processing Systems_. 1410–1418.
  * Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016\.  Text Matching as Image Recognition. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_. 2793–2799.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  GloVe: Global Vectors for Word Representation. In _Empirical Methods in Natural Language Processing_. 1532–1543.
  * Richardson et al. (1995) W. Scott Richardson, Mark C. Wilson, Jim Nishikawa, Robert S. Hayward, and others. 1995\.  The Well-Built Clinical Question: A Key to Evidence-Based Decisions.  _ACP Journal Club_ 123, 3 (1995), A12–3.
  * Robertson and Jones (1976) Stephen E. Robertson and K. Sparck Jones. 1976.  Relevance Weighting of Search Terms.  _Journal of the American Society for Information Science_ 27, 3 (1976), 129–146.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, and others. 1995.  Okapi at TREC-3.  _NIST Special Publication SP_ 109 (1995), 109.
  * Salton and Buckley (1988) Gerard Salton and Christopher Buckley. 1988.  Term-Weighting Approaches in Automatic Text Retrieval.  _Information Processing & Management_ 24, 5 (1988), 513–523.
  * Schardt et al. (2007) Connie Schardt, Martha B. Adams, Thomas Owens, Sheri Keitz, and Paul Fontelo. 2007.  Utilization of the PICO Framework to Improve Searching PubMed for Clinical Questions.  _BMC Medical Informatics and Decision Making_ 7, 1 (2007), 16\.
  * Schneider et al. (2018) Rudolf Schneider, Sebastian Arnold, Tom Oberhauser, Tobias Klatt, Thomas Steffek, and Alexander Löser. 2018.  Smart-MD: Neural Paragraph Retrieval of Medical Topics. In _The Web Conference 2018 Companion_. IW3C2, 203–206.
  * Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Bidirectional Attention Flow for Machine Comprehension.  _5th International Conference on Learning Representations_ (2017).
  * Serrà and Karatzoglou (2017) Joan Serrà and Alexandros Karatzoglou. 2017.  Getting Deep Recommenders Fit: Bloom Embeddings for Sparse Binary Input/Output Networks. In _Proceedings of the Eleventh ACM Conference on Recommender Systems_. ACM, 279–287.
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  Learning Semantic Representations Using Convolutional Neural Networks for Web Search. In _Proceedings of the 23rd International Conference on World Wide Web_. ACM, 373–374.
  * Tellex et al. (2003) Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. 2003.  Quantitative Evaluation of Passage Retrieval Algorithms for Question Answering. In _Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval_. ACM, 41–47.
  * Wan et al. (2016) Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016\.  A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_. 2835–2841.
  * Wang et al. (2017) Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017.  Gated Self-Matching Networks for Reading Comprehension and Question Answering. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. 189–198.
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-End Neural Ad-Hoc Ranking with Kernel Pooling. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 55–64.
  * Yang et al. (2016a) Liu Yang, Qingyao Ai, Jiafeng Guo, and W Bruce Croft. 2016a.  aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. ACM, 287–296.
  * Yang et al. (2016b) Liu Yang, Qingyao Ai, Damiano Spina, Ruey-Cheng Chen, Liang Pang, W Bruce Croft, Jiafeng Guo, and Falk Scholer. 2016b.  Beyond Factoid QA: Effective Methods for Non-Factoid Answer Sentence Retrieval. In _European Conference on Information Retrieval_. Springer, 115–128.
  * Zhang and Zhou (2006) Min-Ling Zhang and Zhi-Hua Zhou. 2006.  Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization.  _IEEE transactions on Knowledge and Data Engineering_ 18, 10 (2006), 1338–1351.
  * Zhu et al. (2019) Ming Zhu, Aman Ahuja, Wei Wei, and Chandan K. Reddy. 2019\.  A Hierarchical Attention Retrieval Model for Healthcare Question Answering. In _The World Wide Web Conference_. ACM, 2472–2482.


# Solitons, peakons and periodic cusp wave solutions for the Fornberg-Whitham
equation

Jiangbo Zhou Corresponding author. Tel.: +86-511-88969336; Fax:
+86-511-88969336<EMAIL_ADDRESS>Lixin Tian  Nonlinear Scientific
Research Center, Faculty of Science, Jiangsu University, Zhenjiang, Jiangsu
212013, China

###### Abstract

In this paper, we employ the bifurcation method of dynamical systems to
investigate the exact travelling wave solutions for the Fornberg-Whitham
equation $u_{t}-u_{xxt}+u_{x}+uu_{x}=uu_{xxx}+3u_{x}u_{xx}$. The implicit
expression for solitons is given. The explicit expressions for peakons and
periodic cusp wave solutions are also obtained. Further, we show that the
limits of soliton solutions and periodic cusp wave solutions are peakons.

###### keywords:

Fornberg-Whitham equation , soliton , peakon , periodic cusp wave solution

,

##  1 Introduction

The Fornberg-Whitham equation

$u_{t}-u_{xxt}+u_{x}+uu_{x}=uu_{xxx}+3u_{x}u_{xx},$ (1.1)

has appeared in the study of qualitative behaviors of wave breaking [1, 2]. It
is a nonlinear dispersive wave equation. Since Eq.(1.1) was derived, little
attention has been paid to studying it. In [3], Fornberg and Whitham obtained
a peaked solution of the form
$u(x,t)=A\exp{(-\frac{1}{2}\left|{x-\frac{4}{3}t}\right|)}$, where $A$ is an
arbitrary constant. In [4], we constructed a type of bounded travelling wave
solutions for Eq.(1.1), which are called kink-like and antikink-like wave
solutions. Unfortunately, the results in [3, 4] are not complete. In the
present paper, we continue to derive more travelling wave solutions for
Eq.(1.1), so that we can supplement the results of [3, 4].

The remainder of the paper is organized as follows. In Section 2, we discuss
the bifurcation curves and phase portraits of travelling wave system. In
Section 3, we obtain the implicit expression for solitons and the explicit
expressions for peakons and periodic cusp wave solutions. At the same time, we
show that the limits of solitons and periodic cusp wave solutions are peakons.
A short conclusion is given in Section 4.

##  2 Bifurcation and phase portraits of travelling wave system

Let $u=\varphi(\xi)$ with $\xi=x-ct$ be the solution for Eq.(1.1); then it
follows that

$-c\varphi^{\prime}+c\varphi^{\prime\prime\prime}+\varphi^{\prime}+\varphi\varphi^{\prime}=\varphi\varphi^{\prime\prime\prime}+3\varphi^{\prime}\varphi^{\prime\prime}.$
(2.1)

Integrating Eq. (2.1) once we have

$\varphi^{\prime\prime}(\varphi-c)=g-c\varphi+\varphi+\frac{1}{2}\varphi^{2}-(\varphi^{\prime})^{2},$
(2.2)

where $g$ is the integral constant.

Let $y=\varphi^{\prime}$; then we get the following planar dynamical system:

$\left\\{{\begin{array}[]{l}\frac{\textstyle d\varphi}{\textstyle d\xi}=y,\\\
\frac{\textstyle dy}{\textstyle d\xi}=\frac{\textstyle
g-c\varphi+\varphi+\frac{1}{2}\varphi^{2}-y^{2}}{\textstyle\varphi-c},\end{array}}\right.$
(2.3)

with a first integral

$H(\varphi,y)=(\varphi-c)^{2}(y^{2}-\frac{1}{4}\varphi^{2}+(\frac{1}{2}c-\frac{2}{3})\varphi+\frac{1}{4}c^{2}-\frac{1}{3}c-g)=h,$
(2.4)

where $h$ is a constant.

Note that (2.3) has a singular line $\varphi=c$. To avoid the line temporarily
we make transformation $d\xi=(\varphi-c)d\zeta$. Under this transformation,
Eq.(2.3) becomes

$\left\\{{\begin{array}[]{l}\frac{\textstyle d\varphi}{\textstyle
d\zeta}=(\varphi-c)y,\\\ \frac{\textstyle dy}{\textstyle
d\zeta}=g-c\varphi+\varphi+\frac{1}{2}\varphi^{2}-y^{2}.\\\
\end{array}}\right.$ (2.5)

System (2.3) and system (2.5) have the same first integral as (2.4).
Consequently, system (2.5) has the same topological phase portraits as system
(2.3) except for the straight line $\varphi=c$. Obviously, $\varphi=c$ is an
invariant straight-line solution for system (2.5).

For a fixed $h$, (2.4) determines a set of invariant curves of system (2.5).
As $h$ is varied, (2.4) determines different families of orbits of system
(2.5) having different dynamical behaviors. Let $M(\varphi_{e},y_{e})$ be the
coefficient matrix of the linearized version of (2.5) at the equilibrium point
$(\varphi_{e},y_{e})$; then

$M(\varphi_{e},y_{e})=\left({{\begin{array}[]{*{20}c}{\quad\quad
y_{e}\hfill}&&&{\varphi_{e}-c\hfill}\\\
{\varphi_{e}-(c-1)\hfill}&&&{-2y_{e}\hfill}\\\ \end{array}}}\right)$ (2.6)

and at this equilibrium point, we have

$J(\varphi_{e},y_{e})=\det
M(\varphi_{e},y_{e})=-2y_{e}^{2}-(\varphi_{e}-c)[\varphi_{e}-(c-1)],$ (2.7)
$p(\varphi_{e},y_{e})=\mathrm{trace}(M(\varphi_{e},y_{e}))=-y_{e}.$ (2.8)

By the theory of planar dynamical systems (see [5]), for an equilibrium point
of a planar dynamical system, if $J<0$, then this equilibrium point is a
saddle point; it is a center point if $J>0$ and $p=0$; if $J=0$ and the
Poincaré index of the equilibrium point is 0, then it is a cusp.

By using the first-integral value and properties of equilibrium points, we
obtain the bifurcation curves as follows:

$g_{1}(c)=\frac{1}{2}(c-1)^{2},$ (2.9)

$g_{2}(c)=\frac{1}{2}(c-1)^{2}-\frac{1}{18},$ (2.10)

$g_{3}(c)=\frac{1}{2}(c-1)^{2}-\frac{1}{2}.$ (2.11)

Obviously, the three curves have no intersection point and
$g_{3}(c)<g_{2}(c)<g_{1}(c)$ for arbitrary constant $c$.

Using the bifurcation method for vector fields (e.g., [5]), we have the
following result which describes the locations and properties of the singular
points of system (2.5).

###### Theorem 2.1

For given any constant wave speed $c\neq 0$, let

$\varphi_{1\pm}=c-1\pm\sqrt{(c-1)^{2}-2g}\quad for\quad g\leq g_{1}(c),$
(2.12) $y_{1\pm}=\pm\sqrt{g-\frac{1}{2}c^{2}+c}\quad for\quad g\geq g_{3}(c).$
(2.13)

Then we have

(1)If $g<g_{3}(c)$, then system (2.5) has two equilibrium points
$(\varphi_{1-},0)$ and $(\varphi_{1+},0)$, which are saddle points.

(2)If $g=g_{3}(c)$, then system (2.5) has two equilibrium points $(c-2,0)$ and
$(c,0)$. $(c-2,0)$ is a saddle point and $(c,0)$ is a cusp.

(3)If $g_{3}(c)<g<g_{2}(c)$, then system (2.5) has four equilibrium points
$(\varphi_{1-},0)$, $(\varphi_{1+},0)$, $(c,y_{1-})$ and $(c,y_{1+})$.
$(\varphi_{1-},0)$ is a saddle point and $(\varphi_{1+},0)$ is a center point
enclosing the orbit which connects the saddle points $(c,y_{1-})$ and
$(c,y_{1+})$.

(4)If $g=g_{2}(c)$, then system (2.5) has four equilibrium points
$(c-\frac{4}{3},0)$, $(c-\frac{2}{3},0)$, $(c,-\frac{2}{3})$ and
$(c,\frac{2}{3})$, which satisfy
$H(c-\frac{4}{3},0)=H(c,-\frac{2}{3})=H(c,\frac{2}{3})$ and form a triangular
orbit which encloses the center point $(c-\frac{2}{3},0)$.

(5)If $g_{2}(c)<g<g_{1}(c)$, then system (2.5) has four equilibrium points
$(\varphi_{1-},0)$, $(\varphi_{1+},0)$, $(c,y_{1-})$ and $(c,y_{1+})$.
$(\varphi_{1+},0)$ is a center point enclosing the orbit which is homoclinic
for the saddle point $(\varphi_{1-},0)$.

(6)If $g=g_{1}(c)$, then system (2.5) has three equilibrium points $(c-1,0)$,
$(c,-\frac{\sqrt{2}}{2})$ and $(c,\frac{\sqrt{2}}{2})$. $(c-1,0)$ is a cusp.
$(c,-\frac{\sqrt{2}}{2})$ and $(c,\frac{\sqrt{2}}{2})$ are two saddle points.

(7)If $g>g_{1}(c)$, then system (2.5) has two equilibrium points $(c,y_{1-})$
and $(c,y_{1+})$. They are saddle points.

The phase portraits of system (2.5) are given in Fig.1.

(a)  (b)

(c)  (d)

(e)  (f)

(g)

Figure 1: The phase portraits of system (2.5). (a) $g<g_{3}(c)$; (b)
$g=g_{3}(c)$; (c) $g_{3}(c)<g<g_{2}(c)$; (d) $g=g_{2}(c)$; (e)
$g_{2}(c)<g<g_{1}(c)$; (f) $g=g_{1}(c)$; (g) $g>g_{1}(c)$.

##  3 Solitons, peakons and periodic cusp wave solutions

Suppose that $\varphi(\xi)(\xi=x-ct)$ is a travelling wave solution for
Eq.(1.1) for $\xi\in(-\infty,+\infty)$, and
$\mathop{\lim}\limits_{\xi\to-\infty}\varphi(\xi)=A$,
$\mathop{\lim}\limits_{\xi\to\infty}\varphi(\xi)=B$, where $A$ and $B$ are two
constants. If $A=B$, then $\varphi(\xi)$ is called a soliton solution. If
$A\neq B$, then $\varphi(\xi)$ is called a kink (or an antikink) solution.
Usually, a soliton solution for Eq.(1.1) corresponds to a homoclinic orbit of
system (2.3) and a periodic travelling wave solution for Eq.(1.1) corresponds
to a periodic orbit of system (2.3). Similarly, a kink (or an antikink) wave
solution of Eq.(1.1) corresponds to a heteroclinic orbit (or the so-called
connecting orbit) of system (2.3). The graphs of the homoclinic orbit,
periodic orbit and their limit cure are shown in Fig.2.

The following lemma gives the relationship of soliton solutions of Eq.(1.1)
and homoclinic orbits of system (2.3).

###### Lemma 3.1

Assume that $\Gamma$ is a homoclinic orbit of system (2.3) and its parameter
expression is $\varphi=\varphi(\xi)$ and $y=y(\xi)$; then $u=\varphi(\xi)$
with $\xi=x-ct$ is a soliton solution for Eq.(1.1).

_Proof._ From Fig.1(e), we can see that the homoclinic orbit $\Gamma$ encloses
$(\varphi_{1+},0)$ and connects $(\varphi_{1-},0)$. Therefore,
$\lim_{|\xi|\rightarrow\infty}\varphi(\xi)=\varphi_{1-}$.

On the other hand, $u=\varphi(\xi)$ is the solution for system (2.3). This
implies that $u=\varphi(\xi)$ is the solution for Eq.(2.1). Thus,
$u=\varphi(x-ct)$ is the soliton solution for Eq.(1.1).

(a)  (b)

(c)

Figure 2: The orbits of system (2.3). (a) The homoclinic orbit (corresponding
to $g_{2}(c)<g<g_{1}(c)$). (b) The limit curve of homoclinic orbit and
periodic orbit (corresponding to $g=g_{2}(c)$). (c) The periodic orbit
(corresponding to $g_{3}(c)<g<g_{2}(c)$).

In Fig.2(a), the homoclinic orbit of system (2.3) can be expressed as

$y=\pm\frac{(\varphi-\varphi_{1-})\sqrt{\varphi^{2}+l_{1}\varphi+l_{2}}}{2(\varphi-c)}\quad
for\quad\varphi_{1-}<\varphi<\varphi_{2+},$ (3.1)

where

$l_{1}=\frac{2}{3}(1-3c-3\sqrt{(c-1)^{2}-2g}),$ (3.2)
$l_{2}=\frac{2}{3}(1-4c+3c^{2}-3g+(3c+1)\sqrt{(c-1)^{2}-2g}),$ (3.3)
$\varphi_{2+}=-\frac{1}{3}(1-3c-3\sqrt{(c-1)^{2}-2g}+2\sqrt{1-3\sqrt{(c-1)^{2}-2g}}).$
(3.4)

Substituting Eq.(3.1) into the first equation of system (2.3) and integrating
along the homoclinic orbits, we have

${\int_{\varphi}^{\varphi_{2+}}{\frac{s-c}{(s-\varphi_{1-})\sqrt{s^{2}+l_{1}s+l_{2}}}ds=-\frac{1}{2}|\xi|}}.$
(3.5)

It follows from (3.5) that

$\beta(\varphi_{2+})=\beta(\varphi)\exp(-\frac{1}{2}|\xi|),$ (3.6)

where

$\beta_{(}\varphi)=\frac{(2\sqrt{\varphi^{2}+l_{1}\varphi+l_{2}}+2\varphi+l_{1})(\varphi-\varphi_{1-})^{\alpha_{1}}}{(2\sqrt{a_{1}}\sqrt{\varphi^{2}+l_{1}\varphi+l_{2}}+b_{1}\varphi+l_{3})^{\alpha_{1}}},$
(3.7) $l_{1}=\frac{2}{3}(1-3c-3\sqrt{(c-1)^{2}-2g}),$ (3.8)
$l_{2}=\frac{2}{3}(1-4c+3c^{2}-3g+(3c+1)\sqrt{(c-1)^{2}-2g}),$ (3.9)
$l_{3}=\frac{4}{3}(2-5c+3c^{2}-6g+(3c+2)\sqrt{(c-1)^{2}-2g}),$ (3.10)
$a_{1}=4(1-2c+c^{2}-2g+\sqrt{(c-1)^{2}-2g}),$ (3.11)
$b_{1}=-\frac{4}{3}-4\sqrt{(c-1)^{2}-2g},$ (3.12)
$\alpha_{1}=-\frac{1+\sqrt{(c-1)^{2}-2g},}{2\sqrt{(c-1)^{2}-2g+\sqrt{(c-1)^{2}-2g}}}.$
(3.13)

(3.6) is the implicit expression for solitons for Eq.(1.1). We show the graphs
of the solitons in Fig.3 under some parameter conditions. From Fig.3, we can
see that when $g_{2}(c)<g<g_{1}(c)$ and $g$ tends to $g_{2}(c)$, the solitons
lose their smoothness and tend to peakons.

(a)  (b)

(c)  (d)

Figure 3: The solitons for Eq.(1.1). (a) $c=2$, $g=0.499999$; (b) $c=2$,
$g=0.49$; (c) $c=2$, $g=0.46$ ; (d) $c=2$, $g=0.45$.

Note the following facts: when $g_{2}(c)<g<g_{1}(c)$ and $g$ tends to
$g_{2}(c)$, the limit curve of such homoclinic orbit of system (2.3) is a
triangle with the following three line segments (see Fig.2(b)):

$y=\pm\frac{1}{2}(\varphi-c+\frac{4}{3})\quad
for\quad\varphi_{1-}\leq\varphi\leq\varphi_{2},$ (3.14)

and

$\varphi=c\quad for\quad-\frac{2}{3}\leq y\leq\frac{2}{3}.$ (3.15)

Let us have $g_{2}(c)<g<g_{1}(c)$ and $g$ tends to $g_{2}(c)$; then we obtain
that

$\varphi(\xi)=\frac{4}{3}\exp({-\frac{1}{2}|\xi|})+c-\frac{4}{3},$ (3.16)

which implies that for arbitrary constant $c\neq 0$, Eq.(1.1) has peakons

$u(x,t)=\frac{4}{3}\exp({-\frac{1}{2}|x-ct|})+(c-\frac{4}{3}).$ (3.17)

Obviously, $u$ has peaks at $x-ct=0$. We show graphs of the peakons in Fig.4
under some parameter conditions.

(a)  (b)

Figure 4: The peakons for Eq.(1.1). (a) $c=-1$ ; (b) $c=2$.

###### Remark 3.1

(1) If we take $c=\frac{4}{3}$ in (3.17), then we can see that (3.17) agrees
with the result in [3].

(2) In the phase portaits, the triangle curve corresponds to a peakon
solution.

We have the following lemma, similar to Lemma 3.1, which indicates the
relationship of periodic wave solutions for Eq.(1.1) and periodic orbits of
system (2.3).

###### Lemma 3.2

Assume that $\Gamma$ is a periodic orbit of system (2.3) and that its
parameter expression is $\varphi=\varphi(\xi)$ and $y=y(\xi)$; then
$u=\varphi(\xi)$ with $\xi=x-ct$ is a periodic wave solution for Eq.(1.1).

In Fig.2(c), the periodic orbit can be expressed as

$y=\pm\sqrt{\frac{1}{4}\varphi^{2}-(\frac{1}{2}c-\frac{2}{3})\varphi-\frac{1}{4}c^{2}+\frac{1}{3}c+g}\quad
for\quad\varphi_{2-}\leq\varphi\leq c,$ (3.18)

and

$\varphi=c\quad for\quad y_{1-}\leq y\leq y_{1+},$ (3.19)

where

$\varphi_{2-}=\frac{1}{3}(-4+3c+\sqrt{2(9c^{2}-18c+8-18g)}).$ (3.20)

Substituting (3.18) into the first equation of system (2.3) and integrating
along the periodic orbit, we have

${\int_{\varphi}^{c}{\frac{1}{\sqrt{\varphi^{2}-(2c-\frac{8}{3})\varphi-c^{2}+\frac{4}{3}c+4g}}}ds=-\frac{1}{2}\xi}\quad
for\quad\xi<0,$ (3.21)

and

${\int_{\varphi}^{c}{\frac{1}{\sqrt{\varphi^{2}-(2c-\frac{8}{3})\varphi-c^{2}+\frac{4}{3}c+4g}}}ds=\frac{1}{2}\xi}\quad
for\quad\xi>0.$ (3.22)

It follows from (3.21) and (3.22) that

$\varphi(\xi)=l_{+}\exp({-\frac{1}{2}|\xi|})+l_{-}\exp({\frac{1}{2}|\xi|})+(c-\frac{4}{3})\quad
for\quad\varphi_{2-}\leq\varphi\leq c,$ (3.23)

where

$l_{\pm}=\frac{1}{6}(4\pm 3\sqrt{4g+4c-2c^{2}}).$ (3.24)

Let

$T=2|\ln(\varphi_{2-}-c+\frac{4}{3})-\ln(2l_{-})|.$ (3.25)

Then

$u(x,t)=\varphi(x-ct-2nT)\quad for\quad(2n-1)T<x-ct<(2n+1)T,$ (3.26)

are periodic cusp wave solutions for Eq.(1.1) with $2T$ period. Clearly, when
$g_{3}(c)<g<g_{2}(c)$ and $g\rightarrow g_{2}(c)$, $T\rightarrow\infty$,
$l_{+}\rightarrow\frac{3}{4}$, $l_{-}\rightarrow 0$, and $u(x,t)$ in (3.26)
tends to

$u(x,t)=\frac{4}{3}\exp({-\frac{1}{2}|x-ct|})+(c-\frac{4}{3}).$ (3.27)

(3.27) is identical with (3.17). The graphs of some periodic waves for
Eq.(1.1) are shown in Fig.5 under some parameter conditions. From Fig.5 we can
see that $g_{3}(c)<g<g_{2}(c)$ and $g$ tends to $g_{2}(c)$, the periodic cusp
wave solutions tend to peakons.

(a)  (b)

(c)  (d)

Figure 5: The periodic cusp wave solutions for Eq.(1.1). (a) $c=2$, $g=0.3$;
(b) $c=2$, $g=0.4$; (c) $c=2$, $g=0.4444444$ ; (d) $c=2$, $g=0.444444444444$.

###### Remark 3.2

(1) In the phase portaits, the semi-elliptic closed curve with one side on the
singular line $\varphi=c$ corresponds to a periodic cusp wave solution.

(2) In [4], we dealt with the case $g\leq g_{1}(c)$, and obtained the kink-
like and antikink-like wave solutions for Eq.(1.1).

##  4 Conclusion

In this work, by using the bifurcation method, we obtain the analytic
expressions for solitons, peakons and periodic wave solutions for the
Fornberg-Whitham equation, given as (3.6), (3.17) and (3.26), respectively. We
also show the relationships among the solitons, peakons and periodic cusp wave
solutions.

## References

  * [1] G. B. Whitham, Variational methods and applications to water wave, Proc. R. Soc. Lond. Ser. A 299 (1967) 6-25.
  * [2] R. Ivanov, On the integrability of a class of nonlinear dispersive wave equations, J. Nonlinear Math. Phys. 1294 (2005) 462-468.
  * [3] B. Fornberg, G. B. Whitham, A numerical and theoretical study of certain nonlinear wave phenomena, Philos. Trans. R. Soc. Lond. Ser. A 289 (1978) 373-404.
  * [4] J. Zhou, L. Tian, A type of bounded traveling wave solutions for the Fornberg-Whitham equation, J. Math. Anal. Appl. 346 (2008) 255-261.
  * [5] D. Luo, et al., Bifurcation Theory and Methods of Dynamical Systems, World Scientific Publishing Co., London, 1997.