# Follow the Timeline! Generating Abstractive and Extractive Timeline Summary
in Chronological Order

Xiuying Chen  Computational Bioscience Reseach Center, King Abdullah
University of Science and Technology<EMAIL_ADDRESS>,  Mingzhe Li
Wangxuan Institute of Computer Technology, Peking University
<EMAIL_ADDRESS>,  Shen Gao  Wangxuan Institute of Computer Technology,
Peking University<EMAIL_ADDRESS>,  Zhangming Chan  Wangxuan Institute of
Computer Technology, Peking University<EMAIL_ADDRESS>,  Dongyan
Zhao  Wangxuan Institute of Computer Technology, Peking University
<EMAIL_ADDRESS>,  Xin Gao  Computational Bioscience Reseach Center, King
Abdullah University of Science and Technology<EMAIL_ADDRESS>,
Xiangliang Zhang  1 University of Notre Dame; 2 King Abdullah University of
Science and Technology<EMAIL_ADDRESS>and  Rui Yan  Gaoling School of
Artificial Intelligence, Renmin University of China<EMAIL_ADDRESS>
###### Abstract.

Nowadays, time-stamped web documents related to a general news query floods
spread throughout the Internet, and timeline summarization targets concisely
summarizing the evolution trajectory of events along the timeline. Unlike
traditional document summarization, timeline summarization needs to model the
time series information of the input events and summarize important events in
chronological order. To tackle this challenge, in this paper, we propose a
_Unified Timeline Summarizer_ (UTS) that can generate abstractive and
extractive timeline summaries in time order. Concretely, in the encoder part,
we propose a graph-based event encoder that relates multiple events according
to their content dependency and learns a global representation of each event.
In the decoder part, to ensure the chronological order of the abstractive
summary, we propose to extract the feature of event-level attention in its
generation process with sequential information remained and use it to simulate
the evolutionary attention of the ground truth summary. The event-level
attention can also be used to assist in extracting summary, where the
extracted summary also comes in time sequence. We augment the previous Chinese
large-scale timeline summarization dataset and collect a new English timeline
dataset. Extensive experiments conducted on these datasets and on the out-of-
domain Timeline 17 dataset show that UTS achieves state-of-the-art performance
in terms of both automatic and human
evaluations111https://github.com/iriscxy/Unified-Timeline-Summarizer.

Timeline Summarization, Extractive Summarization, Abstractive Summarization

††copyright: acmcopyright††journal: TOIS††journalyear: 2022††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††price: 15.00††doi:
10.1145/3517221††ccs: Information retrieval Summarization

##  1\. Introduction

The rapid growth of World Wide Web means that time-stamped document floods
spread throughout the Internet. General search engines simply return web pages
ranked by query relevance, but they are not quite capable of handling
ambiguous intentioned queries, such as a query about evolving news “COVID-19”.
People may have a myriad of general interests about the beginning, the
evolution, or the most up-to-date situation, while simply ranking the returned
webpages according to their relevance is insufficient. In many cases, readers
are tired of navigating every document in the overwhelming collection: they
want to monitor the evolution trajectory of hot topics by simply browsing.
Summarization is an ideal solution to provide a condensed, informative
document reorganization for a faster and better representation of news
evolution. Timeline summary temporally summarizes evolutionary news as a
series of individual but correlated component summaries and hence offers an
option to understand the big picture of a developing situation (Yan et al.,
2011b).

Existing timeline summarization approaches such as (Yan et al., 2011b; Li and
Li, 2013; Ren et al., 2013) are all based on extraction methods. However,
these methods rely on human-engineered features and sophisticated abilities
that are crucial to high-quality summarization, such as paraphrasing,
generalization, or the incorporation of real-world knowledge, which are
possible only in an abstractive framework. Recently, with the emergence of
strong generative neural models for text (Bahdanau et al., 2015), abstractive
techniques are also becoming increasingly popular. Hence, we propose the
_abstractive timeline summarization task_ in our early work (Chen et al.,
2019), which aims to concisely paraphrase the event information in the input
article. An example case is shown in Table 1, where the article consists of
events of a great entertainer in different periods, and the summary correctly
summarizes the important events from the input article in order.

Abstractive summarization approaches including (See et al., 2017; Hsu et al.,
2018; Gavrilov et al., 2019; Zhang et al., 2020a) have been proven to be
useful in traditional summarization task. However, unlike traditional document
summarization, the timeline summarization dataset consists of a series of
time-stamped events, and it is crucial for the timeline summarization model to
capture this time series information to better guide the chronological summary
generation process. Besides, the fidelity problem is also of vital importance
for timeline summarization, where mixing the information of different events
leads to a bad summary. Take the example in Table 1 for example, the bad
summary confuses the birthplace and the residence, the first album, and the
best-selling album of the celebrity. Herein, the good summary is the ground
truth summary from our dataset, and the bad summary is a wrong summary with
typical errors we found in a preliminary experiment. As we found in the
experiment, such infidelity phenomena is a commonly-faced problem in
summarization tasks.

To tackle the above challenges, in our previous work (Chen et al., 2019), we
come up with a _Memory-based Timeline Summarization_ (MTS) model.
Specifically, we first use an event embedding module with selective reading
units to embed all events. Then, we propose a key-value memory module storing
time-series information to guide the summary generation process. Concretely
speaking, the key in the memory module is the time position embedding that
represents the time series information, and the values are the corresponding
event representations. The value item includes local and global
representation, where local value is the output from the event embedding
module, and global value is taken from the average local representation. Keys
together form a timeline and we use the time position of events on the
timeline to guide the generation process. Finally, in each decoding step, we
introduce event-level attention and use it to determine word-level attention
to avoid confusion between events.

Events |  Michael Jackson (dubbed as “King of Pop”) was born on August 29, 1958 in Gary, Indiana. He is the seventh child in his family.
---|---
|  In 1971, Jackson released his first solo “got to be there”, marking the
beginning of his solo career.
|  In late 1982, Jackson’s sixth album, “Thriller”, was released, where videos
”Beat It”, ”Billie Jean” in it are credited with breaking racial barriers and
transforming the medium into an art form and promotional tool.
|  In March 1988, Jackson built a new home named Neverland Ranch in
California, where more than 100 arcade machines were stored here.
|  In 2000, Guinness World Records recognized him for supporting 39 charities
and donated more than 300 million dollars to charities in his own name, more
than any other entertainer.
Bad summary |  Michael Jackson was born on August 29, 1958 in Gary, California. In 1971, his first album “Thriller” was released. In 2000, Guinness World Records recognized him for supporting 39 charities.
Good summary |  Michael Jackson was born on August 29, 1958 in Gary, Indiana.  His sixth album “Thriller” was released in 1982. In 2000, Guinness World Records recognized him for supporting 39 charities.
Table 1. Example of timeline summarization. The text in pink demonstrates time
stamp, and text in blue demonstrates wrong event description. Events are split
by lines.

In MTS, the time information is captured in an implicit and indirect way. MTS
stores the time position embedded in the memory and hopes the decoder will
learn to attend to the correct time position in the training process. However,
that strategy is rather weak supervision, where it is hard to verify and
ensure the decoder indeed captures the time-sequential information. In this
work, we take one step further and improve our previously proposed MTS
framework with explicit timeline guidance modeling. In other words, we
carefully design a strategy that lets the time information be a clear guidance
signal for the summarization process.

Overall, in this paper, we propose a novel _Unified Timeline Summarizer_ (UTS)
that can generate abstractive and extractive timeline summaries in time order.
For the abstractive part, concretely, in the encoder part, we first propose a
graph-based event encoder that relates multiple events according to their
content dependency and learns a representation of each event. The motivation
is that the importance of each event and whether it should be included in the
summary does not only depend on itself but also is related to other events.
Take Table 1 for example, Jackson releases his first solo album might be an
important event, but its importance is weakened by his “Thriller” album that
breaks the racial barriers. Hence, the representation from the graph encoder
incorporates global information from other events, thus is used to replace the
old global representation in the memory. In the decoder part, to avoid the
situation in the bad summary in Table 1, where it confuses the birthplace and
the residence because the model is not sensitive to the timeline, we propose a
summary decoder that emphasizes the time information. Concretely, to ensure
the chronological order of the abstractive summary, we propose to extract the
feature of event-level attention in its generation process with sequential
information remained and use it to simulate the evolutionary attention of the
ground truth summary.

In terms of the extractive part, we present a sentence embedding module to
encode each sentence. Next, a sentence extractor sequential selects important
sentences to be included in the summary. The event-level attention can also be
used to assist in extracting summary in this process, where we devise a time-
aware inconsistency loss function to penalize the inconsistency between
abstractive attention and extractive attention. Note that the extractive
summary is extracted one by one, thus the extracted summary also comes in time
sequence.

We empirically compare MTS and UTS on the public
dataset222https://github.com/yingtaomj/Learning-towards-Abstractive-Timeline-
Summarization proposed by our early work (Chen et al., 2019). This is a large-
scale real-world timeline summarization dataset, which consists of a series of
time-stamped events and the corresponding summary. Moreover, since this
previous dataset only includes a timeline corpus about celebrities, we augment
the dataset with cases about social events. We also collect an English
timeline summarization dataset. Experimental results on these datasets and on
out-of-domain Timeline17 dataset show that our newly proposed UTS model can
significantly outperform the existing methods. Particularly, UTS-abs yields
4.47% and 5.90% percentage point improvement in terms of ROUGE-1 on celebrity
and event timeline datasets compared with our early work MTS. In addition to
the comprehensive evaluation, we also evaluate our proposed graph encoder and
attention mechanism by a fine-grained analysis. The analysis reveals how the
model leverages the explicit timeline information to guide the abstractive and
extractive summarization process and provides us insights on why they can
achieve big improvement over state-of-the-art methods.

Overall, our contributions can be summarized as follows:

$\bullet$ We propose a unified abstractive and extractive timeline
summarization framework, where a time-aware inconsistency loss function is
proposed to unify these two processes.

$\bullet$ We propose a graph-based encoder that relates multiple events
according to their content dependency and learns the global representation of
each event.

$\bullet$ We propose to use the evolutionary attention of the ground truth
summary to guide both the abstractive and extractive summary generation
process, to ensure that the generated summaries follow strict time order.

$\bullet$ We also augment the first real-world large-scale timeline
summarization dataset with social event corpus and corpus in English333Data
will be released in camera-ready version.. Experiments conducted on the three
datasets and the out-of-domain benchmark Timeline 17 dataset show that our
model outperforms all baselines, including state-of-the-art models.
Experiments also verify the effectiveness of each module in UTS as well as its
interpretability.

The rest of the paper is organized as follows: We summarize related work in
§2. We then formulate our research problem in §3 and elaborate our approach in
§4. §5 gives the details of our experimental setup and §6 presents the
experimental results. Finally, §7 concludes the paper.

##  2\. Related Work

We detail related work on text generation methods, timeline summarization,
extractive summarization, abstractive summarization, unified summarization,
and memory network.

###  2.1. Text Generation Methods

In recent years, sequence-to-sequence (seq2seq) (Sutskever et al., 2014) based
neural networks have been proved effective in generating a fluent sentence.
The seq2seq model is originally proposed for machine translation and later
adapted to various natural language generation tasks, such as text
summarization (Wang et al., 2019a; Paulus et al., 2018; Gehrmann et al., 2018;
Lin et al., 2018; Wang et al., 2019b) and dialogue generation (Tao et al.,
2018b; Yao et al., 2017; Cai et al., 2019; Zhao et al., 2020; Zhang et al.,
2020b). Rush et al. (2015) apply the seq2seq mechanism with attention model to
the text summarization field. Then See et al. (2017) add copy mechanism and
coverage loss to generate summarization without out-of-vocabulary and
redundancy words. The seq2seq architecture has also been broadly used in a
dialogue system. Tao et al. (2018a) propose a multi-head attention mechanism
to capture multiple semantic aspects of the query and generate a more
informative response. Yao et al. (2017) propose to use the content introducing
method to solve the problem of generating a meaningless response. Wang et al.
(2018a) use three channels for widening and deepening the topics of interest
and try to make the conversational model chat more turns.

###  2.2. Timeline Summarization

The timeline summarization task is firstly proposed by Allan et al. (2001),
where they define temporal summaries of news stories as extracting a single
sentence from each event within a news topic. Later, a series of works (Yan et
al., 2011b; Yan et al., 2011a, 2012; Zhao et al., 2013) further investigate
timeline summarization task. Yan et al. (2011b) formally formulate the task as
an optimization problem via iterative substitution from a set of sentences to
a subset of sentences that satisfies the above requirements, balancing
coherence/diversity measurement and local/global summary quality. In follow-up
work, Yan et al. (2011a) propose to model trans-temporal correlations among
component summaries for timelines, using inter-date and intra-date sentence
dependencies, and present a novel combination. There are also works focusing
on tweets summarization that is related to timeline summarization. For
example, (Ren et al., 2013) focus on the problem of selecting meaningful
tweets given a user’s interests; the dynamic nature of user interests, the
sheer volume, and the sparseness of individual messages make this a
challenging problem. Specifically, they consider the task of time-aware tweets
summarization, based on a user’s history and collaborative social influences
from “social circles”. Ghalandari and Ifrim (2020) compare different timeline
summarization strategies using appropriate evaluation frameworks. For a more
robust evaluation, they also present a new timeline summarization dataset,
which spans longer time periods than previous datasets. However, all the above
works are based on extractive methods, which are not as flexible as
abstractive approaches.

The most similar work to ours is proposed by (Steen and Markert, 2019), where
they construct a word-adjacency graph, and then generate new sentences from
this graph by finding paths from the sentence start node to the sentence end
node. This is very different from our neural-based approach, and we
demonstrate the superiority of our model in the experiment.

###  2.3. Extractive Summarization

Despite the focus on abstractive summarization, extractive summarization
remains an attractive method. In extractive summarization, Kobayashi et al.
(2015) propose a summarization method using document-level similarity based on
word embeddings. Meanwhile, Filippova et al. (2015) use an RNN to delete words
in a document for sentence compression. Yan and Wan (2015) propose more
meaningful and informative units named frequent deep dependency sub-structure
and a topic-sensitive multi-task learning model for multi-doc summarization.
Cheng and Lapata (2016) propose a general framework for single-document text
summarization using a hierarchical article encoder composed with an attention-
based extractor. Following this, Nallapati et al. (2017b) propose a simple
RNN-based sequence classifier that outperforms or matches the state-of-art
models at the time. Chen et al. (2018) introduce a model which iteratively
polishes the document representation on many passes through the document, so
as to produce a better summary. In another approach, Narayan et al. (2018) use
a reinforcement learning method to optimize the ROUGE evaluation metric for
text summarization. Ren et al. (2018) study the use of sentence relations,
_e.g.,_ contextual sentence relations, title sentence relations, and query
sentence relations, so as to improve the performance of extractive
summarization.

Recently, pre-trained language models are also applied in summarization for
contextual word representations (Zhong et al., 2020; Liu and Lapata, 2019).
Another intuitive structure for extractive summarization is the graph, which
can better utilize the statistical or linguistic information between
sentences. Early works focus on document graphs constructed with the content
similarity among sentences, like LexRank (Erkan and Radev, 2004) and TextRank
(Mihalcea and Tarau, 2004a). Some recent works aim to incorporate a relational
prior into the encoder by graph neural networks (GNNs) (Yasunaga et al.,
2017).

###  2.4. Abstractive Summarization

Recently, with the emergence of strong generative neural models for text
(Bahdanau et al., 2014), abstractive summarization is also becoming
increasingly popular (Nallapati et al., 2017b; See et al., 2017). These models
typically take the form of convolutional neural networks (CNN) or recurrent
neural networks (RNN). For example, Rush et al. (2015) propose an encoder-
decoder model which uses a local attention mechanism to generate summaries.
Nallapati et al. (2016) further develop this work by addressing problems that
had not been adequately solved by the basic architecture, such as keyword
modeling and capturing the hierarchy of sentence-to-word structures. In
follow-up work, Nallapati et al. (2017a) propose a new summarization model
which generates summaries by sampling a topic one sentence at a time, then
producing words using an RNN decoder conditioned on the sentence topic. Zhu et
al. (2020) tackles the cross-lingual summarization task, which aims at
summarizing a document in one language into another language. They propose a
method inspired by the translation pattern in the process of obtaining a
cross-lingual summary. A series of works relies on prototype text to assist in
summarization. Cao et al. (2018a) chose the template with the highest
similarity to the input sentence as a soft template to generate summaries.
Following this, Gao et al. (2019) proposed to generate the summary with
pattern based on prototype editing. Summarization techniques have also been
used in other tasks such as related work generation (Chen et al., 2021) and
headline generation (Li et al., 2021).

###  2.5. Unified Summarization

Unified summarization here means unifying extractive and abstractive
summarization tasks together. It is a common way to propose a multi-task
framework that utilizes the benefits from one task to augment the performance
of the other task. For example, Hsu et al. (2018) proposed a unified framework
that takes advantage of both extractive and abstractive summarization using an
attention mechanism, which is a combination of the sentence-level attention.
Chen and Bansal (2018) introduced a multi-step procedure, namely compression
paraphrase, for abstractive summarization, which first extracts salient
sentences from documents and then rewrites them in order to get final
summaries. Li et al. (2018b) introduced a guiding generation model, where the
keywords in source texts are first retrieved with an extractive model. The
most similar work to ours is (Hsu et al., 2018), where they use sentence-level
attention to modulate the word-level attention such that words in less
attended sentences are less likely to be generated. Their sentence-level
attention is static during the generation process, while in our model, the
high-level attention changes in each decode step depending on the current
generated word which is more reasonable.

###  2.6. Memory Network

The memory network proposed by Sukhbaatar et al. (2015) generally consists of
two components. The first one is a memory matrix to save information (_i.e.,_
memory slots) and the second one is a neural network to read/write the memory
slots. The memory network has shown better performance than traditional long-
short term memory network in several tasks, such as question answering
(Sukhbaatar et al., 2015; Pavez et al., 2018; Ma et al., 2018; Gao et al.,
2018), machine translation (Maruf and Haffari, 2018), text summarization (Kim
et al., 2019; Chen et al., 2019), dialog system (Chu et al., 2018; Wu et al.,
2019), job-resume matching (Yan et al., 2019) and recommendation (Ebesu et
al., 2018; Wang et al., 2018b; Zhou et al., 2019). The reason is that the
memory network can store the information in a long time range and has more
memory storage units than LSTM which has a single hidden state. Following
memory network, there are many variations of memory network have been
proposed, _i.e.,_ key-value memory network (Miller et al., 2016) and dynamic
memory network (Xiong et al., 2016; Kumar et al., 2016). Representative works
include (Gao et al., 2020), where they generate more meaningful answers in
E-commerce question-answering by a read-and-write memory consisting of
selective writing units to conduct reasoning among these reviews.

In our work, we apply the key-value memory network on the timeline
summarization task and fuse it into the generation process.

##  3\. Problem Formulation

Before detailing our answer generation model, we first introduce our notations
listed in Table 2.

Symbol | Description
---|---
$X$ | a document consists of multiple events
$Y$ | ground truth timeline summary
$\hat{Y}$ | generated timeline summary
$x_{i}$ |  $i$-th event in input document
$w^{i}_{j}$ |  $j$-th word in $i$-th event
$T_{e}$ | number of input events
$T_{w}^{i}$ | number of words in $i$-th event
$T_{y}$ | number of words in ground truth summary
$T_{ys}$ | number of sentences in the ground truth timeline summary
$l_{i}$ | extract label for $i$-th sentence in the summary
Table 2. Glossary.

UTS takes a list of events $X=(x_{1},...,x_{T_{e}})$ as inputs, where $T_{e}$
is the number of events. Each event $x_{i}$ is a list of words:
$x_{i}=(w_{1}^{i},w_{2}^{i},...,w_{T_{w}^{i}}^{i})$, where $w_{j}^{i}$ is the
$j$-th word in $i$-th event, and $T_{w}^{i}$ is the word number of event
$x_{i}$.

In the abstractive part, UTS-abs aims to generate a summary
$\hat{Y}=(\hat{y}_{1},...,\hat{y}_{T_{y}})$ that is not only grammatically
correct but also consistent with the event information such as occurrence
place and time. Essentially, UTS-abs tries to optimize the parameters to
maximize the probability $P(Y|X)=\prod_{t=1}^{T_{y}}P(y_{t}|X)$, where
$Y=(y_{1},...,y_{T_{y}})$ is the ground truth summary.

For the extractive part, UTS-ext targets at generating a score vector
$\hat{L}=\\{\hat{l}_{1},\dots,\hat{l}_{T_{ys}}\\}$ for each sentence, where
each score denotes the sentence’s extracting probability. We convert the
human-written summaries to gold label vector $L=\\{l_{1},...,l_{T_{ys}}\\}$,
where $l_{i}\in\\{0,1\\}$ denotes whether the $i$-th sentence is selected (1)
or not (0). During the training process, the cross-entropy loss is calculated
between $L$ and $\hat{L}$, which is minimized to optimize $\hat{L}$.

##  4\. Model

###  4.1. Overview

In this section, we introduce our Unified Timeline Summarizer (UTS) in detail.
The overview of UTS is shown in Figure 1 and can be split into two parts, one
aims to generate an abstractive summary, and one targets selecting important
sentences as a summary.

Abstractive part includes: (1) Event Embedding Module (See § 4.2): To obtain
the vector representations for each event, we employ a recurrent network with
Selective Reading Units (SRU) to learn the local representations. (2) Graph-
based Encoder (See § 4.3): The representations learned in the last module do
not incorporate interaction between events. Hence, we propose a graph-based
encoder to learn the global representation of each event incorporating the
information from other events and the relationship between them. (3) Time-
Event Memory (See § 4.4): we propose a time-event memory, which stores the
local and global event representation, with time position keys together
forming a timeline. (4) Summary Generator (See § 4.5): eventually, we use an
RNN-based decoder to generate the summary under the guidance of event-level
attention and word-level attention.

Extractive part includes: (5) Sentence Embedding Module (See § 4.6): this
module embeds the sentence to a vector representation in a similar way to the
event embedding module. (6) Sentence Extractor (See § 4.7): the sentence
extractor selects the salient sentences as the summary following the
sequential time order.

Additionally, we propose (7) Chronological-Attention Unifier (See § 4.8), to
let the two parts complement each other by unifying the attention
distributions of abstractive parts and extractive parts. Concretely, we
propose a time-aware inconsistency loss to penalize the inconsistency between
these two tasks.

Although some encoder and decoder modules in MTS are similar to UTS, there are
three significant differences in our UTS model compared with MTS:

  1. (1)

MTS encodes each event independently, without considering the information
interaction between events. While in UTS, we propose a graph encoder, which
learns global representations for input events.

  2. (2)

We propose a unified timeline framework that can not only generate an
abstractive summary, but also an extractive summary. That is, only UTS
includes the extractive part.

  3. (3)

We propose to unify the abstractive and extractive parts together, where the
two tasks can benefit each other.

Specifically, we show the comparison between MTS and UTS in Table 3.

Table 3. Comparision between MTS and UTS. | MTS | UTS
---|---|---
Event Embedding Module | SRU | SRU
Graph-based Encoder | - | Transformer
Time-Event Memory | Key-Value Memory | Key-Value Memory
Summary Generator | Editing Gate | Editing Gate
Sentence Embedding Module | - | SRU
Sentence Extractor | - | RNN
Unifier | - | Inconsistency loss
Figure 1.  Overview of UTS. We divide our model into abstractive summarization
part and extractive summarization part. Abstractive part includes: (1) Event
Embedding Module, (2) Graph-based encoder, (3) Time-Event Memory, and (4)
Summary Generator. Extractive part includes: (5) Sentence Embedding Module and
(6) Sentence Extractor. Additionally, there is a (7) Chronological Attention
Unifier that unifies the two tasks.

###  4.2. Event Embedding Module

We first propose an event embedding module to obtain the word-level and event-
level vector representations. To begin with, we use an embedding matrix $e$ to
map a one-hot representation of each word in $x_{i}$ into a high-dimensional
vector space. We denote $e(w^{i}_{t})$ as the embedding representation of word
$w^{i}_{t}$. We then employ a bi-directional recurrent neural network (Bi-RNN)
to model the temporal interactions between words:

(1) $\displaystyle\overleftarrow{h^{i}_{t}}$
$\displaystyle=\text{LSTM}^{\text{enc}}([e(w^{i}_{t});p^{i}],\overleftarrow{h^{i}_{t-1}}),$
(2) $\displaystyle\overrightarrow{h^{i}_{t}}$
$\displaystyle=\text{LSTM}^{\text{enc}}([e(w^{i}_{t});p^{i}],\overrightarrow{h^{i}_{t-1}}),$
(3) $\displaystyle h^{i}_{t}$
$\displaystyle=\overrightarrow{h^{i}_{t}}+\overleftarrow{h^{i}_{t}},$

where “;” denotes the concatenation between vectors, and $h^{i}_{t}$ denotes
the hidden state of $t$-th word in Bi-RNN for event $x_{i}$. To capture the
sequential information of events, we randomly initialize a _time position
encoding_ vector $p^{i}$ of $i$-th event to be included in the Bi-RNN input.

Apart from obtaining word representation $h^{i}_{t}$, we also need to gain
event representation. Simply taking the final state of Bi-RNN
$h^{i}_{T^{i}_{w}}$ as the representation of the whole event cannot fully
capture the feature of the whole event. Thus, we employ the selective reading
module consisted of SRU proposed in (Chen et al., 2018) to gain new event
representation $a^{i}$:

(4) $\displaystyle s_{t}^{i}$
$\displaystyle=\text{SRU}(s_{t-1}^{i},[h_{t}^{i},h^{i}_{T^{i}_{w}}]),$ (5)
$\displaystyle a^{i}$ $\displaystyle=s^{i}_{T_{w}^{i}},$

where $s_{t}^{i}$ is the hidden state of $t$-th SRU cell in $i$-th event. At
the high level, SRU is a modified version of GRU, which replaces the update
gate in original GRU (Cho et al., 2014) with a new gate taking each input
$h_{t}^{i}$ and coarse event representation $h^{i}_{T_{w}}$ into
consideration. We omit the details here due to limited space and readers can
refer to (Chen et al., 2018) for details. So far, we obtain the representation
of $i$-th event $a^{i}$ and $t$-th word in $a^{i}$, _i.e._ , $h^{i}_{t}$.

###  4.3. Graph-based Encoder

The event representation $a^{i}$ in the previous section is calculated
independently, without considering the information flow between different
events. However, the importance of each event and whether it should be
included in the summary does not only depend on itself but also is related to
other events. For example, in Table 1, Jackson releases his first solo album
might be an important event, but its importance is weakened by his “Thriller”
album that breaks the racial barriers. Hence, we propose a graph-based encoder
to learn the relationship between events and obtain a global event
representation that incorporates such information.

As shown in Figure 1, to embed relationship information, we set up the
relation edges in our document modeling graph. The relation edge in our graph
is firstly initialized by the event representation:

(6) $\displaystyle r^{i,j}=\text{MLP}_{a}([a^{i};a^{j}]),$

where MLP is a multi-layer perceptron.

Next, during the relation-aware encoding process, we incorporate the relation
edge $r^{i,j}$ into the final event representation by self attention
operation:

(7) $\displaystyle b^{i}=\text{RE}(a^{i},a^{*},r^{i,*}),$

where $*$ denotes all indexes between $1$ and ${T_{e}}$. This module is based
on Transformer. Thus, we first introduce Transformer:

(8) $\displaystyle b^{i^{\prime}}=\text{Transformer}(a^{i},a^{*}).$

Concretely, the first input is for query and the second input is for keys and
values. Each output element, $b^{i^{\prime}}$, is computed as weighted sum of
a linearly transformed input values:

(9) $\displaystyle
b^{i^{\prime}}=\sum_{j=1}^{T_{e}}\alpha^{i,j}_{g}\left(a^{j}W^{V}\right).$

Each weight coefficient, $\alpha^{i,j}_{g}$, is computed using a softmax
function:

(10)
$\displaystyle\alpha_{g}^{i,j}=\frac{\exp\left(\beta^{i,j}_{g}\right)}{\sum_{k=1}^{T_{e}}\exp\left(\beta^{i,k}_{g}\right)}.$

$\beta^{i,j}_{g}$ is computed using a compatibility function that compares two
input elements:

(11)
$\displaystyle\beta^{i,j}_{g}=\frac{\left(a^{i}W^{Q}\right)\left(a^{j}W^{K}\right)^{T}}{\sqrt{d}},$

where $d$ is the hidden dimension, and
$W^{Q},W^{K},W^{V}\in\mathbb{R}^{d\times d}$ are parameter matrices.

RE is similar to Transformer, with two changes in Equation 9 and 11.
Specifically, we modify Equation 9 to propagate edge information to the sub-
layer output:

(12) $\displaystyle
b^{i}=\sum_{j=1}^{T_{e}}\alpha^{i,j}_{g}\left(a^{j}W^{V}_{r}+r^{i,j}\right).$

In this way, the representation of each event is more comprehensive,
consisting of its relation dependency information with other events. In the
meantime, when deciding the weight of each edge, i.e., $\beta^{i,j}_{g}$, we
also incorporate relation edge information, since close relationships can have
a great impact on edge weight. Concretely, Equation 11 is changed to:

(13)
$\displaystyle\beta^{i,j}_{g}=\frac{\left(a^{i}W^{Q}_{r}\right)\left(a^{j}W^{K}_{r}+r^{i,j}\right)^{T}}{\sqrt{d}}.$

The intuition for Transformer architecture is that each input is not isolated,
and its representation depends on other inputs as well. In our augmented
Transformer, i.e., graph-based encoder, the polished event representation
$b^{i}$ follows the same idea and expands the dependency between input
documents. $b^{i}$ does not only depend on its corresponding content but also
depends on other inputs, as well as the relationships with others.

###  4.4. Time-Event Memory

As stated in the Introduction, in the timeline dataset, the generated summary
should capture the time-series information to guide the chronological
generation process. Hence, we propose a key-value memory module where keys
together form a timeline, and this time series information is used to guide
the generation process as shown in Figure 2.

The key in this memory is the time position encoding $p^{i}$ introduced in §
4.2. We will use this key as time guidance to extract information from the
value part in the memory, which will be introduced in detail in § 4.5. The
value part stores event information of local aspect in local value and global
aspect in global value. Local value simply stores the event representation
$a^{i}$, which means that only captures information from the current event. On
the other hand, the global value is responsible for learning the event feature
from a global perspective, not only based on itself but also its relationship
with other events. Hence, it stores the graph-based encoder output, $b^{i}$.

###  4.5. Summary Generator

To generate a consistent and informative summary, so as to avoid mixing
information from different time stamps due to unawareness of correct timeline,
we propose an RNN-based decoder that incorporates outputs of time-event memory
module and event representation as illustrated in Figure 2.

Figure 2.  An overview of the summary generator in the abstractive part. The
summary generator generates the next word based on word-level and event-level
attention, as well as the key-value memory.

Following (Li et al., 2018a), we randomly initialize an LSTM cell taking the
concatenation of all event representations as input, and use the output as
decoder initial state:

(14) $\displaystyle h^{\prime}_{0}$
$\displaystyle=\text{LSTM}^{\text{ini}}\left(h_{c},[a^{1};...;a^{T_{e}}]\right),$

where $h_{c}$ is a random variable.

####  4.5.1. Word-level attention

Next, following traditional attention mechanism in (Bahdanau et al., 2015), we
summarize the input document into context vector $c^{\prime}_{t-1}$
dynamically, and the $t$-th decoding step is calculated as:

(15) $\displaystyle h^{\prime}_{t}$
$\displaystyle=\text{LSTM}^{\text{dec}}\left(h^{\prime}_{t-1},[c^{\prime}_{t-1};e(y_{t-1})]\right),$

where $h^{\prime}_{t}$ is the hidden state of $t$-th decoding step. Context
vector $c^{\prime}_{t-1}$ is calculated as:

(16) $\displaystyle\alpha^{t^{\prime}}_{i,j}$
$\displaystyle=W_{a}^{\intercal}\tanh\left(W_{b}h^{\prime}_{t-1}+W_{h}h^{i}_{j}\right),$
(17) $\displaystyle\alpha^{t}_{i,j}$
$\displaystyle=\exp\left(\alpha^{t^{\prime}}_{i,j}\right)/\sum^{T_{e}}_{k=1}\left(\sum^{T^{i}_{w}}_{j=1}\exp\left(\alpha^{t^{\prime}}_{k,j}\right)\right),$
(18) $\displaystyle c^{\prime}_{t-1}$
$\displaystyle=\sum_{i=1}^{T_{e}}\left(\sum_{j=1}^{T^{i}_{w}}\alpha^{t}_{i,j}h^{i}_{j}\right),$

where we first use the decoder state $h^{\prime}_{t-1}$ to attend to each
states $h^{i}_{j}$ which results in the attention distribution
$\alpha^{t}_{i,j}$, shown in Equation 17. $h^{i}_{j}$ denotes the
representation of $j$-th word in event $x_{i}$. Then we use the attention
distribution $\alpha^{t}_{i,j}$ to obtain the weighted sum of document states
as the context vector $c_{t-1}^{\prime}$.

Context vector $c^{\prime}_{t-1}$ here only takes the word-level attention
into consideration without considering event-level information. However, in
timeline summarization, it is important for the model to be aware of which
event it is currently describing, or it may confuse information from different
events and result in an unfaithful summary. Hence, we introduce an event-level
attention $\beta$ similar to the calculation of word-level attention and use
it to adjust word-level attention:

(19) $\displaystyle\beta^{t^{\prime}}_{i}$
$\displaystyle=W_{c}^{\intercal}\tanh\left(W_{d}h^{\prime}_{t-1}+W_{e}a^{i}\right),$
(20) $\displaystyle\beta^{t}_{i}$
$\displaystyle=\exp\left(\beta^{t^{\prime}}_{i}\right)/\sum^{T_{e}}_{j=1}\exp\left(\beta^{t^{\prime}}_{j}\right),$
(21) $\displaystyle\gamma^{t}_{i,j}$
$\displaystyle=\alpha^{t}_{i,j}\beta^{t}_{i}.$

The new context vector $c_{t}$ (replacing $c_{t}^{\prime}$ in Equation 15) is
now calculated as:

(22) $\displaystyle c_{t}$
$\displaystyle=\sum_{i=1}^{T_{e}}\left(\sum_{j=1}^{T^{i}_{w}}\gamma^{t}_{i,j}h^{i}_{j}\right).$

####  4.5.2. Event-level attention

Apart from using event-level attention to directly guide word-level attention,
we also use it to obtain the weighted sum of event representation to be
concatenated in the projection layer in Equation 31:

(23) $\displaystyle e_{t}=\sum_{i=1}^{T_{e}}{\beta^{t}_{i}a^{i}}.$

####  4.5.3. Memory guidance

So far, we have finished the calculation of the context vectors. Next, we
introduce how to incorporate the guidance from memory. We first use hidden
state $h^{\prime}_{t}$ to attend to each key in memory. As stated in § 4.4,
keys, _i.e._ , time position embeddings, conform the timeline that represents
the time series information. Thus, we let the model take advantage of this
sequential information, and calculate the relevance between position encoding
and current state as time-attention $\pi(p^{i},h^{\prime}_{t})$:

(24)
$\displaystyle\pi(p^{i},h^{\prime}_{t})=\exp(h^{\prime}_{t}W_{e}p^{i})/\sum^{T_{e}}_{j=1}\exp(h^{\prime}_{t}W_{e}p^{j}).$

Time-attention is then used to gain the weighted sum of local value $v_{1}$
and global value $v_{2}$ in the memory:

(25) $\displaystyle m^{1^{\prime}}_{t}$
$\displaystyle=\sum^{T_{e}}_{i=1}\pi(p^{i},h^{\prime}_{t})v^{i}_{1},$ (26)
$\displaystyle m^{2^{\prime}}_{t}$
$\displaystyle=\sum^{T_{e}}_{i=1}\pi(p^{i},h^{\prime}_{t})v^{i}_{2}.$

$m^{1^{\prime}}_{t}$ and $m^{2^{\prime}}_{t}$ stores information from
different level, thus should play different roles in generator.

By a fusion gate, local value $m^{1^{\prime}}_{t}$ is changed to $m^{1}_{t}$
and will be incorporated into the projection layer in Euqation 31.

(27) $\displaystyle g^{1}_{t}$
$\displaystyle=W_{o}([h^{\prime}_{t};c_{t};m^{1^{\prime}}_{t}]),$ (28)
$\displaystyle m^{1}_{t}$ $\displaystyle=g^{1}_{t}\cdot m^{1^{\prime}}_{t}.$

We place the local value in the projection layer since $m^{1}_{t}$ stores the
detailed information rather than the global feature in the input, thus should
play an important part when generating each word.

As for the global value $m^{2^{\prime}}_{t}$, it stores the global feature of
the event in a different position, thus should influence the whole generation
process. Concretely, information from $m^{2^{\prime}}_{t}$ is fusioned into
the decoding state $h_{t}^{\prime}$ by a gate:

(29) $\displaystyle g^{2}_{t}$
$\displaystyle=W_{n}([h_{t}^{\prime};c_{t};m^{2^{\prime}}_{t}]),$ (30)
$\displaystyle h_{t}^{\prime}$ $\displaystyle=g^{2}_{t}\cdot
h_{t}^{\prime}+(1-g^{2}_{t})\cdot m^{2^{\prime}}_{t}.$

Finally, an output projection layer is applied to get the final generating
distribution $P_{v}$ over vocabulary:

(31) $\displaystyle
P_{v}=\text{softmax}\left(W_{v}[m^{1}_{t};h^{\prime}_{t};c_{t};e_{t}]+b_{v}\right).$

We concatenate the output of decoder LSTM $h^{\prime}_{t}$, the word context
vector $c_{t}$, the event context vector $e_{t}$, and memory vector
$m^{1}_{t}$ as the input of the output projection layer.

In order to handle the out-of-vocabulary (OOV) problem, we equip the pointer
network (Gu et al., 2016; See et al., 2017) with our decoder, which enables
the decoder capable of copying words from the source text. The design of the
pointer network is the same as the model used in (See et al., 2017), thus we
omit this procedure due to limited space.

Our objective function in the abstractive part is the negative log likelihood
of the target word $y_{t}$, shown in Equation 32:

(32) $\displaystyle\mathcal{L}^{\text{abs}}$
$\displaystyle=-\sum^{T_{y}}_{t=1}\log P_{v}(y_{t}).$

The gradient descent method is employed to update the parameters in the
abstractive part to minimize this loss function.

###  4.6. Sentence Embedding Module

So far, we introduce the abstractive timeline summarization part. Next, we
will introduce the extractive summarization part in UTS, and how to unify
these two tasks.

Our sentence embedding module takes inspiration from (Chen et al., 2018),
where the embedding module also takes the form of a hierarchical structure and
consists of an iterative polishing process to better encoder the input
document. Concretely, we employ a new Bi-RNN to process each sentence and
obtain the representation $\hat{h}_{t}^{i}$, denoting the $t$-th word in
$i$-th sentence. We use the last hidden state to represent the overall
sentence representation, denoted as $\hat{h}^{i}_{T_{w}}$. The document
representation is initialized as the average of all sentence representations:

(33) $\displaystyle
D_{1}=\tanh\left(W\frac{1}{T_{ys}}\sum_{i=1}^{T_{ys}}\left[\hat{h}^{i}_{T_{w}}\right]+b\right).$

Next, to model the sequential relationship between sentences and obtain a more
comprehensive sentence representation, we iteratively polish the sentence and
document representations. For brevity, we take the first iteration as an
example to illustrate the process. Concretely, there is an RNN based on SRU
(introduced in Equation 4) in the iteration:

(34) $\displaystyle\hat{a}_{1}^{i}$
$\displaystyle=\text{SRU}(\hat{a}_{0}^{i-1},[\hat{h}^{i-1}_{T_{w}},D_{1}]),$
(35) $\displaystyle D_{2}$
$\displaystyle=\text{GRU}_{\text{iter}}(\hat{a}_{1}^{T_{ys}},D_{1}),$

where $\hat{a}^{i}_{1}$ is the hidden state of $i$-th SRU cell in the first
iteration. In this way, we iteratively polish the sentence and document
representation. We use $I$ to denote the iteration number, thus the final
representation for $i$-th sentence is $\hat{a}_{I}^{i}$.

Figure 3.  An overview of the sentence extractor in the extractive part. In
each decoding step, a sentence is to be included in the summary in sequence.

###  4.7. Sentence Extractor

Different from previous work that builds a classifier to assign importance
score to each sentence, we use an RNN consisting of LSTM cells to select
sentences, wherein each step a sentence is selected as illustrated in Figure
3. Following traditional attention mechanism in (Bahdanau et al., 2015), we
summarize the input document sentences into context vector $c_{\text{ext}}$
dynamically, and the $t$-th decoding step is calculated as:

(36) $\displaystyle\tilde{h}_{t+1}$
$\displaystyle=\text{LSTM}^{\text{ext}}(\tilde{h}_{t},[c_{t}^{\text{ext}};\hat{a}^{ot}_{I}]),$
(37) $\displaystyle c_{t}^{\text{ext}}$
$\displaystyle=\sum^{T_{ys}}_{i=1}\hat{\beta}_{t}^{i}\hat{a}^{i}_{I},$ (38)
$\displaystyle ot$ $\displaystyle=\text{argmax}(\text{MLP}(\tilde{h}_{t})),$

where $ot$ is the index of the selected in $t$ step, and $\hat{a}^{ot}_{I}$ is
the hidden state of the previously selected sentence. $\hat{\beta}_{t}^{i}$ is
the attention weight on $i$-th sentence in $t$-th step, and is computed in a
similar way to §4.5.1. Thus, the details are omitted here due to limited
space.

In this way, our extracted summary is generated in sequence, so as to better
capture the sequential information in the input:

(39) $\displaystyle\mathcal{L}^{\text{ext}}$
$\displaystyle=-\sum_{i=1}^{T_{ys}}\log P_{s}\left(l_{i}\right).$

###  4.8. Chronological-Attention Unifier

In both abstractive and extractive timeline summarization tasks, the attention
on the input document should both follow the time sequential order. Hence, it
is intuitive to encourage these two levels of attention to be mostly
consistent with each other during training as an intrinsic learning target for
free (_i.e.,_ without additional human annotation).

Figure 4. The illustration of the Chronological-Attention Unifier. After a
convolution and a tile operation, the event-level attention in the summary
generation is compared with the sentence-level attention in sentence
extractor, where a novel inconsistency loss function is introduced to penalize
the inconsistency between these two levels of attentions.

In §4.5.1, we propose the event-level attention $\beta$ in abstractive part,
while in §4.7, the extractor pays sentence-level attention $\hat{\beta}$ on
the input. The event-level attention evolves each time a new word is predicted
in the summary generator, while the sentence-level attention evolves when a
new sentence is selected. Hence, we first use a convolutional neural network
(CNN) to extract the evolving event attention feature from the generator.
Concretely, as shown in Figure 4, a convolution along the decode step number
axis is conducted on the event-level attention map, and the new attention
matrix with the sentence-numbered axis is obtained. Then, for each sentence-
select step, we duplicate the event-level attention $\beta^{i}_{t}$ multiple
times, where the duplicate number is the sentence number in the $i$-th event.

Finally, we would like the event-level attention to be high when the sentence-
level attention is high. Hence, we design the following time-aware
inconsistency loss:

(40)
$\displaystyle\mathcal{L}^{\text{inc}}=-\frac{1}{T_{y}}\sum_{t=1}^{T_{ys}}\log\left(\frac{1}{|\mathcal{K}|}\sum_{t\in\mathcal{K}}\hat{\beta}_{t}\times\beta_{t}\right),$

where $\mathcal{K}$ is the set of top K attended sentences. This implicitly
encourages the distribution of the sentence-level attentions to be sharp and
event-level attention to be high. To avoid the degenerated solution for the
distribution of sentence attention to be one-hot and event attention to be
high, we include the original loss functions for training the extractor
($\mathcal{L}^{\text{ext}}$ in §4.7) and abstracter
($\mathcal{L}^{\text{abs}}$ in §4.5). Note that this module is the only part
that the extractor is interacting with the abstracter. Our time-aware
inconsistency loss facilitates our end-to-end trained unified model to be
mutually beneficial to both the extractor and abstracter.

##  5\. Experimental Setup

###  5.1. Research Questions

We list seven research questions that guide the experiments: RQ1 (See § 6.1):
What is the overall performance of UTS? Does it outperform other baselines on
multilingual datasets? RQ2 (See § 6.2): What is the performance of our model
on out-of-domain classic timeline summarization dataset? RQ3 (See § 6.3): What
is the effect of each module in UTS? Does our multi-task framework help better
summarization performance? RQ4 (See § 6.4): Is the time position embedding
useful so that the summary generator can attend to correct information in the
time-event memory? RQ5 (See § 6.5): Can event-level attention correctly guide
word-level attention in decoding process in the abstractive part? RQ6 (See §
6.6): Are the chronological attentions successfuly unified in the abstractive
and extractive summarization tasks? RQ7 (See § 6.7): What is the influence of
the parameter settings?

###  5.2. Dataset

To our best knowledge, there are no large-scale summarization datasets for
timeline. Hence, in our previous work (Chen et al., 2019), we collect a large-
scale timeline dataset from the world’s largest Chinese
encyclopedia444https://baike.baidu.com/. The character subsection of this
website consists of celebrities at all times and in all countries or lands. On
each website page, there is a timeline summary for each character, and in the
character experience section of this page, each event is set as a paragraph
with explanation and details, which is selected as an input article. In the
previous timeline works (Yan et al., 2011a), they did not pre-select important
sentences from the event news articles as a way to test the summarization
ability of the proposed model. Hence, in our work, we did not preprocess the
event paragraph as well, since these event paragraphs are similar to news
articles in content and in style. We filter out irrelevant content such as
cited sources and figures. We did not specifically extract time information
from the input, because our model learns the time information in an implicit
way, instead of particularly encoding it. In total, the training dataset
amounts to 169,423 samples with 5,000 evaluation and 5,000 test samples. On
average, there are 353.79 words and 61.19 words in the article and summary
respectively.

Furthermore, in this work, we first augment the previous dataset with event
timeline summarization cases. On the Chinese encyclopedia, there is also a
social event subsection that includes the developments of related events over
time. Concretely, in the development history section of each page, there are
event paragraphs that describe the development of the corresponding event, and
there is also a corresponding timeline summary for these events. After the
same cleaning operation, we have 83,188 training cases, 3,000 validation, and
3,000 test samples. On average, there are 495.19 words and 141.62 words in the
article and summary respectively.

Note that the above two datasets are both in the Chinese language. To test the
performance of our model on multi-lingual datasets, we collect an English
timeline summarization dataset from Wikipedia websites. Since there are no
character or event subsections in Wikipedia, we filter timeline pages by
checking if there are multiple timestamps in the summary and document on each
website. Other preprocesses are similar to the Chinese encyclopedia. A human
evaluation on 200 sampled cases from the collected corpus shows that 196 cases
are timeline document-summary pairs, 145 of which are about characters, and 51
are about events.

Since we have large-scale English summarization datasets, we can test the
generalization ability of our model on classic timeline summarization
datasets, which are small-scale. Concretely, we report the performance of UTS
on out-of-domain dataset Timeline 17 (TL17) (Tran et al., 2013). TL17 contains
human-written timelines about topics such as civil wars or the British
Petroleum oil disaster, collected from major news outlets. Each topic also has
a set of related news articles scraped from the web.

Datasets | # docs (train/val/test) | avg. document length | avg. summary length | vocabulary size
---|---|---|---|---
words | sentences | words | sentences | document | summary
TL17 | 4,650 | 1,252.33 | 63.76 | 43.66 | 2.73 | 102,099 | 6,725
Celebrity TS | 169,423/5,000/5,000 | 353.79 | 12.76 | 61.19 | 3.97 | 444,725 | 191,334
Event TS | 83,188/3,000/3,000 | 495.19 | 18.73 | 141.62 | 6.05 | 1,083,249 | 368,619
Wiki TS | 140,000/5,000/5,000 | 606.65 | 27.79 | 79.19 | 7.89 | 1,029,617 | 438,011
Table 4. Comparison of summarization datasets with respect to overall corpus
size, size of training, validation, and test set, average document (source)
and summary (target) length (in terms of words and sentences), and vocabulary
size on both on source and target. TS denotes Timeline Summarization.

The statistics of the four datasets are listed in Table 4. We also give
timeline statistic information in Table 5. It can be seen that compared with
TL17 dataset, our three datasets are significantly larger. This again
demonstrates the necessity of our datasets, which are large enough to train a
neural-based model. In terms of timeline-related attributes, the summaries in
our datasets have more date stamps in each sentence on average, which requires
the summarization model to be more time-aware. The average date number in the
document input is smaller in our datasets, this is because that our input
document is shorter than TL17. However, the average sentences/dates ratio of
our datasets is comparable to TL17, proving the time attribute of our
datasets.

Datasets | Document | Summary length | Compression
---|---|---|---
Avg dates | Avg sents/dates | Avg dates | Avg sents/dates | Sent | Date
TL17 | 77.55 | 1.22 | 1.61 | 1.14 | 34.70 | 32.46
Celebrity TS | 11.03 | 1.18 | 3.15 | 1.10 | 3.76 | 3.50
Event TS | 13.09 | 1.54 | 5.55 | 1.19 | 3.03 | 2.36
Wiki TS | 6.81 | 3.99 | 2.49 | 3.17 | 3.44 | 2.74
Table 5. Timeline-specific statistic attributes of our datasets and TL17
dataset.

###  5.3. Comparison Methods

We first conduct an ablation study to prove the effectiveness of each module
in UTS. Then, to evaluate the performance of our proposed dataset and model,
we compare it with the following baselines:

Abstractive baselines:

(1) Pointer-Gen (See et al., 2017) is an RNN based model with an attention
mechanism and allows the system to copy words from the source text via
pointing for abstractive summarization.

(2) FTSum leverages open information extraction and dependency parse
technologies to extract actual fact descriptions from the source text (Cao et
al., 2018b). Since there is no open information extraction tool in Chinese, we
use POS tagging to extract entities and verbs to replace them.

(3) Unified is a unified model combining the strength of extractive and
abstractive summarization proposed in (Hsu et al., 2018), where sentence-level
attention is used to modulate the word-level attention such that words in less
attended sentences are less likely to be generated.

(4) GPG is a model proposed by Shen et al. (2019) which generates summaries by
“editing” pointed tokens instead of hard copying. The editing is performed by
transforming the pointed word vector into a target space with a learned
relation embedding.

(5) SAGCopy is an augmented Transformer with a self-attention guided copy
mechanism, which was proposed by Xu et al. (2020). Specifically, they first
identify the importance of each source word based on the degree centrality
with a directed graph built by the self-attention layer in the Transformer.
They then use the centrality of each source word to guide the copy process
explicitly.

(6) MTS is the first abstractive timeline summarization framework proposed in
our previous work (Chen et al., 2019). This method achieves state-of-the-art
performance on the celebrity timeline summarization dataset.

Extractive baselines:

(1) Lead3 is an extractive baseline that concatenates the first-3 sentences of
each source document as a summary.

(2) TextRank (Mihalcea and Tarau, 2004b) is an unsupervised algorithm while
sentence importance scores are computed based on eigenvector centrality within
weighted-graphs for extractive sentence summarization.

(3) ITS One of state-of-the-art extractive summarization models proposed in
(Chen et al., 2018). ITS iteratively polishes the document representation on
many passes through the document, so as to extract better summaries.

For testing our models on out-of-domain dataset, we compare with a number of
traditional timeline summarization baselines:

(1) Chieu (Chieu and Lee, 2004) is an unsupervised baseline based on direct
summarization.

(2) Martschat (Martschat and Markert, 2018) greedily selects a combination of
sentences from the entire collection, which maximizes submodular functions for
content coverage, textual and temporal diversity and a high count of date
references.

(3) Tran(Binh Tran et al., 2013) is an original date-wise timeline
summarization approach, using regression for both date selection and
summarization, and using all sentences of a date as candidate sentences.

(4) Pubcount (Ghalandari and Ifrim, 2020) is a simple date-wise baseline that
uses the publication count to rank dates, and all sentences published on a
date for candidate selection.

(5) Datawise (Ghalandari and Ifrim, 2020) uses supervised date selection, PM-
MEAN for candidate selection and CENTROID-OPT for summarization.

(6) Clust (Ghalandari and Ifrim, 2020) uses DATEMENTIONCOUNT to rank clusters,
and CENTROID-OPT for summarization.

The performance of these baselines are consistent with the result from
(Ghalandari and Ifrim, 2020).

###  5.4. Evaluation Metrics

For evaluation metrics, we adopt ROUGE F1 score in (Lin, 2004) which is widely
applied for summarization evaluation (Sun et al., 2018; Chen et al., 2018).
The ROUGE metrics compare the generated summary with the reference summary by
computing overlapping lexical units, including ROUGE-1 (unigram), ROUGE-2 (bi-
gram), and ROUGE-L (longest common subsequence).

For the out-of-domain test dataset, we follow (Ghalandari and Ifrim, 2020),
and use the specific timeline evaluation metric, _i.e.,_ Alignment-based ROUGE
F1-score, and Date F1-score. Alignment-based ROUGE F1-score compares the
textual overlap between a system and a ground-truth timeline, while also
considering the assignments of dates to texts. Date F1-score compares only the
dates of a system and a ground-truth timeline.

(Schluter, 2017) notes that only using the ROUGE metric to evaluate
summarization quality can be misleading. Therefore, we also evaluate our model
by human evaluation. Three highly educated participants are asked to score 100
randomly sampled summaries generated by GPG, SAGCopy, MTS, and UTS-abs.
Statistical significance of observed differences between the performance of
two runs are tested using a two-tailed paired t-test and is denoted using ▲(or
▼) for strong significance for $\alpha=0.01$.

###  5.5. Implementation Details

We implement our experiments in TensorFlow (Abadi et al., 2016) on NVIDIA GTX
1080 Ti GPU. For all experiments, our model has 256-dimensional hidden states
and 128-dimensional word embeddings. Following See et al. (2017), we do not
pretrain the word embeddings, instead, they are learned from scratch during
training. We use a vocabulary of 50k words for both source and target. For
time-event memory, the dimension of the key, global value, and local value are
128, 512, and 256 respectively. We initialize all of the parameters randomly
using a uniform distribution in [-0.02, 0.02]. The batch size is set to 16,
and the event number is set to 8. For the abstractive part, during training
and at test time we truncate the article to 400 tokens and limit the length of
the summary to 70 tokens. For the extractive part, we used a greedy algorithm
similar to (Nallapati et al., 2017b) to obtain an oracle summary for each
document to train extractive models. The algorithm generates an oracle
consisting of multiple sentences which maximize the ROUGE-2 score against the
gold summary. We limit the input sentence number to 24, the length of each
input sentence to 20, and the number of selected sentences to 4. For the
chronological-attention unifier, we set $K$ to 3 for computing
$\mathcal{L}^{\text{inc}}$. We use Adagrad optimizer (Duchi et al., 2010) as
our optimizing algorithm and the learning rate is 0.15. (This was found to
work best of Stochastic Gradient Descent, Adadelta, Momentum, Adam, and
RMSProp). We use gradient clipping with a maximum gradient norm of 2, but do
not use any form of regularization. We use loss on the validation set to
implement early stopping. In decoding, we employ a beam search with beam size
4 to generate a more fluent summary sentence. When testing our model on the
out-of-domain Timeline 17 dataset, for each example with S source input
documents, we take the first 400/S tokens from each source document.

For the training efficiency, it takes about 9.7 hours to train an epoch, and
our model reaches the best performance after only 3 epochs. While for baseline
Pointer-Gen, it takes 7 hours to train an epoch, but it reaches the best
performance after 7 epochs. In particular, our model makes much quicker
progress in the early phases of training. This demonstrates the effectiveness
of our unified model. In terms of testing, it takes 1.06 hours to generate
summaries for all the cases in the test dataset. We selected the top-3
checkpoints based on the evaluation loss on the validation set, and report the
averaged results on the test set.

##  6\. Experimental Results

###  6.1. Overall Performance

For research question RQ1, we examine the performance of our model and
baselines in terms of ROUGE as shown in table 6. Firstly, on the celebrity
timeline dataset, abstractive models outperform extractive models by a
substantial margin on our datasets. We attribute this result to the
observation that the gold summary of this dataset tends to use new expressions
to summarize the original input documents. This demonstrates the necessity of
abstractive timeline summarization approaches. Secondly, we compare our
previous model MTS with recently-proposed baselines including SAGCopy and GPG.
These two baselines obtain lower ROUGE scores on our datasets than MTS, which
demonstrates the effectiveness of our previous model. Finally, based on MTS,
our augmented model UTS-ext and UTS-abs achieves even better performance.

Concretely, for the abstractive part, UTS-abs outperforms SAGCopy by 7.56%,
14.92% and 7.61%, and outperforms MTS by 4.47%, 7.68% and 3.95% in terms of
ROUGE-1, ROUGE-2 and ROUGE-L respectively on celebrity timeline dataset. For
the extractive part, our extractive method achieves about 2.23% points
improvement on ROUGE-2 compared with ITS on the celebrity timeline dataset. We
attribute the improvement to two aspects: Firstly, the abstractive objective
can promote the recognition of important sentences for the extractive model
with the chronological attention unifier network. Besides, while extractive
gold label sequences are obtained by greedily optimizing ROUGE-2 on the gold-
standard summary, gold labels may not be accurate. Joint learning of two
objectives may correct some biases for the extractive model due to the
inaccurate labels. The above results prove the superiority of our model. Note
that we mainly compare our model with ITS, because our extractive part is
mostly based on ITS. Our framework can be applied to other extractive models,
and theoretically, will bring benefits for both tasks. We leave it as future
work.

Models | Celebrity Timeline Dataset | Event Timeline Dataset | Wiki Timeline Dataset
---|---|---|---
RG-1 | RG-2 | RG-L | RG-1 | RG-2 | RG-L | RG-1 | RG-2 | RG-L
_Sentence extraction methods_ |  |  |  |  |  |
Lead3 | 32.36 | 17.96 | 30.99 | 21.47 | 9.26 | 15.73 | 25.35 | 5.94 | 20.56
TextRank | 32.27 | 15.34 | 30.86 | 23.89 | 10.43 | 16.66 | 24.98 | 5.47 | 22.40
ITS | 34.03 | 18.20 | 31.24 | 27.94 | 14.28 | 20.39 | 27.82 | 5.91 | 25.37
Unified-ext | 34.18 | 18.29 | 31.16 | 28.06 | 14.39 | 20.47 | 26.48 | 5.82 | 24.28
UTS-ext | 34.81 | 22.26 | 32.03 | 29.12 | 16.01 | 23.06 | 29.00 | 6.64 | 25.81
_Abstractive methods_ |  |  |  |  |  |
Pointer-Gen | 36.61 | 21.35 | 34.51 | 22.56 | 7.84 | 21.00 | 23.12 | 5.07 | 19.65
FTSum | 37.84 | 21.47 | 35.37 | 23.41 | 6.95 | 21.66 | 24.08 | 5.80 | 20.05
Unified-abs | 38.24 | 21.95 | 36.42 | 23.58 | 7.93 | 21.95 | 24.34 | 5.84 | 20.37
GPG | 38.43 | 21.59 | 36.38 | 22.38 | 7.77 | 20.81 | 24.71 | 5.80 | 20.85
SAGCopy | 38.64 | 20.84 | 36.41 | 23.40 | 7.95 | 21.72 | 26.00 | 5.84 | 22.01
MTS | 39.78 | 22.24 | 37.69 | 23.89 | 8.38 | 21.97 | 26.68 | 5.88 | 23.18
UTS-abs | 41.56 | 23.95 | 39.18 | 25.30 | 9.63 | 23.28 | 27.71 | 5.92 | 24.62
Table 6. RQ1: ROUGE scores comparison between baselines. Models and baselines in the top half are extractive, while those in the bottom half are abstractive. All our ROUGE scores have a 95% confidence interval of at most $\pm$0.24 as reported by the official ROUGE script. | Fluency | Informativeness | Fidelity
---|---|---|---
GPG | 2.590 | 2.530 | 2.390
SAGCopy | 2.640 | 2.57 0 | 2.430
MTS | 2.710 | 2.580 | 2.610
UTS-abs | 2.77▲ | 2.62▲ | 2.65▲
Table 7. RQ1: Human evaluation comparison with main baselines on celebrity
timeline dataset.

Our human evaluation study assessed the overall quality of the summaries on
the celebrity timeline dataset by asking three highly educated participants to
rank them taking into account the following criteria: Fluency (is the summary
fluent and grammatical?), Informativeness (does the summary convey important
facts about the topic in question?), and Fidelity (is the summary faithful to
the input?). We pick SAGCopy and GPG as baselines since their performance is
relatively high compared to other baselines. The rating score ranges from 1 to
3 and 3 is the best. The results are presented in Table 7. We can see that our
model performs much better than all baselines. In the fluency indicator, our
model achieves a high score of 2.77, which is higher than 2.59 of GPG and 2.64
of SAGCopy, indicating that our model can reduce the grammatical errors and
improve the readability of the summary. In the informativeness indicator, our
model is 0.05 better than SAGCopy. It indicates that our model can effectively
capture salient information. In the fidelity indicator, UTS-abs outperforms
all baselines by a large margin, which indicates the multi-granularity
semantic information and joint learning with extractive summarization does
help to avoid the unfaithful information of the generated summary. It is worth
noticing that the infidelity problem is a serious problem existing in timeline
summarization, and MTS and UTS-abs greatly alleviates such problem. We also
conduct the paired student t-test between our model and SAGCopy (the row with
shaded background), and the result demonstrates the significance of the above
results. The kappa statistics is 0.46 and 0.49 respectively, which indicates
moderate agreement between annotators555(Landis and Koch, 1977) characterize
kappa values $<$ 0 as no agreement, 0-0.20 as slight, 0.21-0.40 as fair,
0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1 as almost perfect
agreement.. To prove the significance of these results, we also conduct the
paired student t-test between our model and SAGCopy. We obtain a p-value of
$3\times 10^{-8}$, $8\times 10^{-12}$, and $9\times 10^{-11}$ for fluency,
informativeness, and fidelity, respectively.

In 1981, James Cameron directed his first film, “Piranha II”, which was shot
entirely in Italy. Cameron didn’t get along well with an Italian speaking
staff, and the producers didn’t let him participate in the final editing of
the film… In 1984, Cameron released his first self-made and self-directed film
“Terminator”, which costs only 6.5 million dollars… In 1986, James Cameron’s
second self-made work, “Alien 2”, was published… In 1987, “Alien 2” won seven
Academy Award nominations… James Cameron won the best director award at the
14th Saturn Awards for this film. In 1989, Cameron wrote and directed his
third film, “The Abyss”… In 1991, his film “Terminator 2” made 200 million
dollars at the box office in the United States, and he also won the 18th
Saturn Awards for best director and best screenwriter for this film. In 1997,
Cameron directed the film “Titanic”, which wins 1.84 billion at the box
office, and starred Leonardo DiCaprio and Kate Winslet. In 1998, the film won
14 nominations and 11 awards at the 70th Academy Awards. In 2000, he directed
and supervised the TV series “The dark angel of the last world” with the theme
of gene therapy.
---
reference |  In 1981, he directed the first film “Piranha II”. In 1984, he became famous for his science fiction film “Terminator”. In 1986, he wrote and directed the film “Alien 2”. In 1991, he won the best director award and best screenwriter award at the 18th Saturn awards for his film “Terminator 2”. In 1997, his film “Titanic” won 1.84 billion dollars at the box office, breaking the global box office record; it won 11 awards including best picture at the 70th Academy Awards, and James Cameron won the best director award at the Oscars.
GPG |  In 1981, James Cameron directed the film “Piranha II”. Cameron Cameron didn’t get along well with an Italian speaking staff, and the producers didn’t let him participate in the final editing of the film. In 1984, he became famous for his science fiction film “Terminator”. In 1986, he wrote and directed the film “Alien 2”. In 1986, he wrote and directed the film “Alien 2”. In 1991, he won the best director award and best screenwriter award at the 18th Saturn Awards for his film “Terminator 2”. In 1997, his film Titanic won 1.84 billion US dollars at the box office.
SAGCopy |  In 1981, James Cameron directed his first work, “Piranha II”. In 1984, Cameron released his first film, “Terminator”. In 1986, James Cameron wrote and directed his second work, “Alien 2”. In 1997, James Cameron directed the film “Titanic”. In 2000, he directed and supervised the TV series “The dark angel of the last world” with the theme of gene therapy.
UTS-ext |  In 1981, James Cameron directed his first film, “Piranha II”, which was shot entirely in Italy. In 1984, Cameron released his first self-made and self-directed film “Terminator”, which cost only 6.5 million dollars. In 1986, James Cameron’s second self-made work, “Alien 2”, was published. in 1997, Cameron directed the film “Titanic”, which wins 1.84 billion at the box office, and starred Leonardo DiCaprio and Kate Winslet.
UTS-abs |  In 1981, Cameron directed his first work, “Piranha II”. In 1984, he released his first film “Terminator”. In 1986, his second film, “Alien 2”, was published. In 1987, “Alien 2” won the 14th Saturn Award for best director. In 1991, his film “Terminator 2” made 200 million dollars in the United States. In 1997, he directed the film “Titanic”. In 1998, the film won 14 Academy Awards nominations and 11 of them at the 70th Academy Awards.
Table 8. RQ1: Examples of the generated answers by UTS-abs, UTS-ext and
baselines (translated version).
1981年，詹姆斯·卡梅隆执导了第一部作品《食人鱼2》，影片完全在意大利拍摄。卡梅隆和一口意大利语的工作人员相处得并不愉快，而拍摄完毕后制片方不让他参与影片的最终剪辑.
1984年，卡梅隆推出了他第一部自编自导的影片《终结者》，这部影片的拍摄只花了650万美元… 1986年，詹姆斯·卡梅隆自编自导的第二部作品《异形2》问世…
1987年，《异形2》获得了七项奥斯卡奖提名…詹姆斯·卡梅隆凭借此片获得了第14届土星奖最佳导演奖… 1989年，卡梅隆自编自导了第三部电影《深渊》…
1991年，他执导的电影《终结者2》在美国上映后取得了2亿美元的票房，他也凭借该片获得了第18届土星奖最佳导演奖以及最佳编剧奖…
1997年，詹姆斯·卡梅隆执导了电影《泰坦尼克号》，该片获得18.4亿美元的票房，由莱昂纳多·迪卡普里奥、凯特·温斯莱特等主演.
1998年，在第70届奥斯卡金像奖上这部影片获得了14个奥斯卡奖的提名并获得了其中的11个奖项…
2000年，他执导并监制了以基因治疗为题材的电视剧《末世黑天使》…
---
reference |  1981年，詹姆斯卡梅隆执导首部电影《食人鱼2》。 1984年，因自编自导科幻电影《终结者》成名。 1986年，自编自导电影《异形2》。 1991年，凭借电影《终结者2》获得第18届土星奖最佳导演奖以及最佳编剧奖。 1997年，他执导的电影《泰坦尼克号》取得了18.4亿美元的票房，打破全球影史票房纪录;该片在第70届奥斯卡金像奖上获得了包括最佳影片在内的11个奖项，詹姆斯·卡梅隆凭借该片获得了奥斯卡奖最佳导演奖。
GPG |  1981年，詹姆斯·卡梅隆执导了部作品《食人鱼2》，卡梅隆卡梅隆和一口意大利语的工作人员相处得并不愉快，而拍摄完毕后制片方不让他参与影片的最终剪辑. 1984年，卡梅隆凭借科幻电影《终结者》出名。 1986年，他自编自导了电影《异形2》。 1986年，他自编自导了电影《异形2》。 1991年，他凭借《终结者2》获得了第18届土星奖最佳导演奖和最佳编剧奖。 1997年，他的电影《泰坦尼克号》在美国获得了18.4亿票房。
SAGCopy |  1981年，詹姆斯·卡梅隆执导了第一部作品《食人鱼2》。 1984年，卡梅隆推出了他第一部自编自导的影片《终结者》。 1986年，詹姆斯·卡梅隆自编自导的第二部作品《异形2》。 1997年，詹姆斯·卡梅隆执导了电影《泰坦尼克号》。 2000年，他执导并监制了以基因治疗为题材的电视剧《末世黑天使》。
UTS-ext |  1981年，詹姆斯·卡梅隆执导了第一部作品《食人鱼2》，影片完全在意大利拍摄。 1984年，卡梅隆推出了他第一部自编自导的影片《终结者》.这部影片的拍摄只花了650万美元。 1986年，詹姆斯·卡梅隆自编自导的第二部作品《异形2》问世。 1997年，詹姆斯·卡梅隆执导了电影《泰坦尼克号》，该片获得18.4亿美元的票房，由莱昂纳多·迪卡普里奥、凯特·温斯莱特等主演。
UTS-abs |  1981年，卡梅隆执导了第一部作品《食人鱼2》。 1984年，卡梅隆推出了他第一部自编自导的影片《终结者》。 1986年，自编自导的第二部作品《异形2》问世。 1987年，《异形2》获得了第14届土星奖最佳导演奖。 1991年，他执导的电影《终结者2》在美国上映后取得了2亿美元的票房。 1997年，执导了电影《泰坦尼克号》。 1998年，在第70届奥斯卡金像奖上这部影片获得了14个奥斯卡奖的提名并获得了其中的11个奖项。
Table 9. RQ1: Examples of the generated answers by UTS-abs, UTS-ext and
baselines.

We also show a case study in Table 9 with translated version in Table 8
selected from celebrity timelime dataset. The case is about James Cameron’s
career as a director. We omit unimportant information in the input document
due to limited space. The input document includes most of his works, and the
detailed information of each event, while the summary reference only
introduces the main event of his experience, omitting those details and
unimportant events. It can be seen that the summary generated by UTS-abs
successfully captures the important events, and introduces them in the correct
order. The output of our UTS-ext has a high overlap with the ground truth. As
for baseline GPG, it fails to capture the most important events, but includes
irrelevant information such as details in filming “Piranha II”. For baseline
SAGCopy, it also generates unimportant descriptions including information “The
dark angel of the last world”. Moreover, our extractive and abstractive
summary show consistent behavior with the high overlap, which further
indicates that the two methods can jointly promote the recognition of
important information. Compared with the extracted summary, the generated
summary is more concise and coherent.

###  6.2. Out of Domain Test

Next, we address research question RQ2. In Table 10, we present the
performance of UTS on the classic timeline summarization TL17 dataset as an
out-of-domain test. It can be seen that both of our models outperform existing
baselines. Specifically, UTS-ext outperforms the best baseline Datawise by
19.4% on AR1-F score, demonstrating the effectiveness of the neural network in
the traditional extractive style. UTS-abs performs similar to UTS-ext,
improving the AR1-F score of Pubcount by 3.79. This demonstrates that the
abstractive methods can be adapted to out-of-domain small-scale datasets.
Specifically, since our original WikiTS dataset is in encyclopedia style,
while Timeline 17 is a news dataset, this demonstrates that our model can be
applied to datasets of different language styles.

| AR1-F | AR2-F | Date-F1
---|---|---|---
Chieu | 6.66 | 1.9 | 25.1
Martschat | 10.5 | 3.0 | 54.4
Tran | 9.4 | 2.2 | 51.7
Pubcount | 10.5 | 2.7 | 48.1
Datewise | 12.0 | 3.5 | 54.4
Clust | 8.2 | 2.0 | 40.7
UTS-ext | 16.73 | 4.08 | 54.9
UTS-abs | 14.29 | 3.51 | 54.6
Table 10. RQ2: ROUGE scores on out-of-domain TL17 summarization dataset.

###  6.3. Ablation Study

| ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
UTS-abs | 41.56 | 23.95 | 39.18
without multitask | 39.58 | 22.54 | 37.55
without global | 38.66 | 22.87 | 36.76
without local | 39.14 | 23.15 | 36.00
UTS-ext | 34.81 | 22.26 | 32.03
without multitask | 33.78 | 21.09 | 29.03
without global | 33.00 | 18.89 | 27.07
without local | 33.28 | 20.98 | 29.69
Table 11. RQ3: ROUGE scores of different ablation models.

Figure 5. RQ3: Visualizations of time-attention. The figure in the left part
is the attention map in the first decoding step, and the figure in the right
part is in the final decoding step.

Next, we turn to research question RQ3, where we perform an ablation study on
the test set to investigate the influence of different modules in our proposed
UTS model. Modules are tested in four ways: (1) we remove the sentence
extractor and only train the generator to verify the effectiveness of joint
learning on the abstractive summarization; (2) we remove the summary generator
part and only train the sentence extractor to verify the effectiveness of
joint learning on the extractive summarization; (3) we remove the graph-based
encoder and only stores the local representation in the memory to verify the
effectiveness of global representation; (4) we remove the time-event memory
entirely to verify the importance of global and local representation further.

Table 11 presents the results. We find that the ROUGE-2 score of extractive
summarization drops by 5.26% after the summary generator is removed. This
indicates that the joint learning method helps extractive summarization to
benefit from abstractive summarization. ROUGE-2 score of abstractive
summarization drops by 5.54% after the sentence extractor is removed. This
indicates that extractive summarization does help abstractive summarization
identify important sentences during the interactive decoding phrase. ROUGE-2
score of extractive summarization drops by 4.72%, while the ROUGE-2 score of
abstractive summarization drops by 6.25% after the global representation is
removed. It indicates establishing the graph-based encoder to simulate the
relationships between events is necessary to improve the performance of both
extractive and abstractive summarization. ROUGE-2 score drops by 4.72% and
3.45% compared with UTS-abs after removing the global representation and the
local representation. It indicates the semantic information of the time-event
memory is of great importance to encode multiple events.

###  6.4. Analysis of Time Position Embedding

We then address RQ4. The usefulness of time position embedding is reflected by
time-attention in the memory, denoted as $\pi$ as introduced in Equation 24.
If the time position embedding successfully encodes the time information, then
the time-attention should obey the development of the input document. We
visualize the attention map of two randomly sampled examples as shown in
Figure 5 from the celebrity timeline dataset. The figure on the left is the
attention map in the first decoding step, and the figure on the right is in
the final decoding step. The darker the color is, the higher the attention is.
Due to limited space, we omit the corresponding event descriptions. When
decoding starts, UTS-abs learns to pay attention to the first two events,
which always consist of parallel information such as the birthplace and birth
date of the character. The attentions on the last several events are low since
it does not need this information in advance. When decoding ends, UTS-abs
focuses more on the last several events. However, it also pays attention to
the first few events, since timeline summarization is a process of information
accumulation, and later sentences should consider previous information. The
above example demonstrates the effectiveness of time position embedding.

Figure 6. RQ5: Visualizations of two level attentions. The figure above is the
event-level attention and the three figures below are the word-level
attentions of first lead three events.

###  6.5. Analysis of Event-level Attention

We now turn to RQ5, whether event-level attention can guide word-level
attention in the abstractive part. We first conduct a case study to visualize
the two-level attention, as shown in Figure 6. The figure above is the event-
level attention, and the three figures below are word-level attention
corresponding to the first three events. We only show the first 11 words in an
event. The result shows that the third event is the most important event in
this decoding step, and the weights of the words in this event are also
greater than other words on average. The above observation demonstrates that
event-level attention gives the correct guidance for word-level attention.

Apart from the visualization, we also conduct a quantitative analysis to
measure how greatly the word-level attention is influenced by event-level
information, which is reflected by inconsistency loss. We adjust the
inconsistency loss proposed in §4.8 to evaluate the inconsistency between
event attention and word attention. The new consistency loss at $t$-th
decoding step is the negative log-likelihood of the product of attention value
of most attended words and their corresponding event-level attention. The
intuition is to verify whether the event-level attention is high too when
word-level attention is high. When training starts, the inconsistency loss is
around 5.3, and when training ends, the loss drops to 2.1. This means that
event-level information greatly influences the word-level attention and the
model learns to unify these two attentions. We did not directly add
inconsistency loss to training because we found that made UTS perform worse.
Instead, we let the model learn by itself to unify these two attentions.

Figure 7. RQ6: Time-aware inconsistency loss curve.

###  6.6. Analysis of the Unified Chronological Attentions

We then address RQ6, examining whether the chronological attentions in the
abstractive and extractive parts are indeed unified. Remember that we come up
with a time-aware inconsistency loss to unify the two attentions, thus, by
looking at the loss curve, we can examine the effectiveness of this part.

The loss curve of the inconsistency is shown in Figure 7. We can see that when
the training begins, the inconsistency loss fluctuates from time to time,
probably because the model aims to train the extractor and generator
separately at the beginning of the process. However, the average of the
inconsistency loss presents a falling tendency, which means that the extractor
and generator unify during the whole training procedure. In the end, the time-
aware inconsistency loss drops from 4.0 to 2.5.

###  6.7. Robustness of Parameter Setting

Figure 8.  Performance of UTS-abs with different parameter settings.

Finally, we turn to address RQ7 to investigate the robustness of parameter
setting. We train our model in different parameter settings as shown in Figure
8. The hidden size of the RNN is tuned from 64 to 512, and we use the ROUGE
$F_{1}$ score to evaluate each model. As the hidden size grows larger from 64
to 256, the performance rises along with. The increment of hidden size
improves the ROUGE-1 and ROUGE-L scores by 0.54 and 0.77 score. When the
hidden size continuously goes larger from 256 to 512, the performance is
declined slightly. The increment of hidden size leads to a 1.15% and 1.25%
drop in terms of ROUGE-1 and ROUGE-L respectively. Nonetheless, we can find
that each metric is maintained at a stable interval, which demonstrates that
our UTS is robust in terms of different parameter sizes.

##  7\. Conclusion and Future Work

In our previous work, we propose a framework named MTS which aims to generate
summaries that concisely summarize the evolution trajectory along the
timeline. However, in this method, the time information is captured in an
implicit and indirect way, where it is hard to verify and ensure the decoder
indeed captures the time-sequential information. Hence, in this work, we
propose a novel _Unified Timeline Summarizer_ (UTS) that can generate
abstractive and extractive timeline summaries in time order. Specifically, in
the encoder part, we propose a graph-based event encoder that relates multiple
events according to their content dependency and learns a representation of
each event. In the decoder part, to ensure the chronological order of the
abstractive summary, we propose to extract the feature of event-level
attention in its generation process with sequential information remained and
use it to simulate the evolutionary attention of the ground truth summary. The
event-level attention can also be used to assist in extracting summary, where
we devise a time-aware inconsistency loss function to penalize the
inconsistency between abstractive attention and extractive attention. Note
that the extractive summary is generated one by one, thus the extracted
summary also comes in time sequence. We augment the character timeline
summarization dataset proposed in our previous work with the event timeline
summarization corpus and English corpus. Experimental results on these
datasets and on out-of-domain Timeline 17 dataset show that our UTS model can
significantly outperform the existing methods. In the near future, we aim to
propose a multi-modal time-aware timeline summarization framework.

## Acknowledgments

We would like to thank the anonymous reviewers for their constructive
comments. This work was supported by National Key Research and Development
Program of China (No. 2020YFB1406702), National Natural Science Foundation of
China (NSFC Grant No. 62122089 & No. 61876196)

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: a system for large-scale machine learning.. In _OSDI_ , Vol. 16. 265–283.
  * Allan et al. (2001) James Allan, Rahul Gupta, and Vikas Khandelwal. 2001.  Temporal summaries of new topics. In _SIGIR_. ACM, 10–18.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Neural Machine Translation by Jointly Learning to Align and Translate. In _ICLR_.
  * Binh Tran et al. (2013) Giang Binh Tran, Mohammad Alrifai, and Dat Quoc Nguyen. 2013\.  Predicting relevant news events for timeline summaries. In _Proceedings of the 22nd International Conference on World Wide Web_. 91–92.
  * Cai et al. (2019) Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. 2019\.  Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework. In _EMNLP_.
  * Cao et al. (2018a) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018a.  Retrieve, rerank and rewrite: Soft template based neural summarization. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 152–161.
  * Cao et al. (2018b) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018b.  Faithful to the original: Fact aware neural abstractive summarization. In _AAAI_.
  * Chen et al. (2021) Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. 2021.  Capturing Relations between Scientific Papers: An Abstractive Model for Related Work Section Generation. In _ACL_.
  * Chen et al. (2019) Xiuying Chen, Zhangming Chan, Shen Gao, Meng-Hsuan Yu, Dongyan Zhao, and Rui Yan. 2019\.  Learning towards Abstractive Timeline Summarization. In _IJCAI_.
  * Chen et al. (2018) Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, and Rui Yan. 2018\.  Iterative Document Representation Learning Towards Summarization with Polishing.  _EMNLP_ (2018).
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting.  _ACL_ (2018).
  * Cheng and Lapata (2016) Jianpeng Cheng and Mirella Lapata. 2016.  Neural summarization by extracting sentences and words.  _arXiv preprint arXiv:1603.07252_ (2016).
  * Chieu and Lee (2004) Hai Leong Chieu and Yoong Keok Lee. 2004.  Query based event extraction along a timeline. In _Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval_. 425–432.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _EMNLP_ (2014).
  * Chu et al. (2018) Eric Chu, Prashanth Vijayaraghavan, and Deb Roy. 2018\.  Learning Personas from Dialogue with Attentive Memory Networks. In _EMNLP_.
  * Duchi et al. (2010) John C. Duchi, Elad Hazan, and Yoram Singer. 2010\.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.  _JMLR_ 12 (2010), 2121–2159.
  * Ebesu et al. (2018) Travis Ebesu, Bin Shen, and Yi Fang. 2018.  Collaborative Memory Network for Recommendation Systems. In _SIGIR_.
  * Erkan and Radev (2004) Günes Erkan and Dragomir R Radev. 2004.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ 22 (2004), 457–479.
  * Filippova et al. (2015) Katja Filippova, Enrique Alfonseca, Carlos A Colmenares, Łukasz Kaiser, and Oriol Vinyals. 2015\.  Sentence compression by deletion with lstms. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. 360–368.
  * Gao et al. (2018) Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. 2018\.  Motion-Appearance Co-Memory Networks for Video Question Answering. In _CVPR_.
  * Gao et al. (2019) Shen Gao, Xiuying Chen, Piji Li, Zhangming Chan, Dongyan Zhao, and Rui Yan. 2019\.  How to Write Summaries with Patterns? Learning towards Abstractive Summarization through Prototype Editing.  _arXiv preprint arXiv:1909.08837_ (2019).
  * Gao et al. (2020) Shen Gao, Xiuying Chen, Zhaochun Ren, Dongyan Zhao, and Rui Yan. 2020.  Meaningful Answer Generation of E-Commerce Question-Answering.  _arXiv preprint arXiv:2011.07307_ (2020).
  * Gavrilov et al. (2019) Daniil Gavrilov, Pavel Kalaidin, and Valentin Malykh. 2019\.  Self-Attentive Model for Headline Generation. In _European Conference on Information Retrieval_. Springer, 87–93.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018\.  Bottom-Up Abstractive Summarization. In _EMNLP_.
  * Ghalandari and Ifrim (2020) Demian Gholipour Ghalandari and Georgiana Ifrim. 2020.  Examining the State-of-the-Art in News Timeline Summarization. In _ACL_.
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016\.  Incorporating Copying Mechanism in Sequence-to-Sequence Learning.  _CoRR_ abs/1603.06393 (2016).
  * Hsu et al. (2018) Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018\.  A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss. ACL, 132–141.
  * Kim et al. (2019) Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019.  Abstractive Summarization of Reddit Posts with Multi-level Memory Networks. In _NAACL_.
  * Kobayashi et al. (2015) Hayato Kobayashi, Masaki Noguchi, and Taichi Yatsuka. 2015\.  Summarization based on embedding distributions. In _Proceedings of the 2015 conference on empirical methods in natural language processing_. 1984–1989.
  * Kumar et al. (2016) Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016.  Ask Me Anything: Dynamic Memory Networks for Natural Language Processing.  _ArXiv_ abs/1506.07285 (2016).
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ (1977), 159–174.
  * Li et al. (2018b) Chenliang Li, W. Xu, S. Li, and Sheng Gao. 2018b.  Guiding Generation for Abstractive Text Summarization Based on Key Information Guide Network. In _NAACL-HLT_.
  * Li and Li (2013) Jiwei Li and Sujian Li. 2013.  Evolutionary hierarchical dirichlet process for timeline summarization. In _ACL_ , Vol. 2. 556–560.
  * Li et al. (2018a) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2018a.  A Survey on Deep Learning for Named Entity Recognition.  _arXiv preprint arXiv:1812.09449_ (2018).
  * Li et al. (2021) Mingzhe Li, Xiuying Chen, Min Yang, Shen Gao, Dongyan Zhao, and Rui Yan. 2021\.  The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglement. In _AAAI_.
  * Lin (2004) Chin-Yew Lin. 2004\.  Rouge: A package for automatic evaluation of summaries.  _Text Summarization Branches Out_ (2004).
  * Lin et al. (2018) Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. 2018\.  Global Encoding for Abstractive Summarization. In _ACL_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  _arXiv preprint arXiv:1908.08345_ (2019).
  * Ma et al. (2018) Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. 2018.  Visual Question Answering With Memory-Augmented Networks. In _CVPR_.
  * Martschat and Markert (2018) Sebastian Martschat and Katja Markert. 2018.  A Temporally Sensitive Submodularity Framework for Timeline Summarization. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_. 230–240.
  * Maruf and Haffari (2018) Sameen Maruf and Gholamreza Haffari. 2018.  Document Context Neural Machine Translation with Memory Networks. In _ACL_.
  * Mihalcea and Tarau (2004a) Rada Mihalcea and Paul Tarau. 2004a.  Textrank: Bringing order into text. In _Proceedings of the 2004 conference on empirical methods in natural language processing_. 404–411.
  * Mihalcea and Tarau (2004b) Rada Mihalcea and Paul Tarau. 2004b.  TextRank: Bringing Order into Text. In _EMNLP_.
  * Miller et al. (2016) Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016\.  Key-Value Memory Networks for Directly Reading Documents.  _ArXiv_ abs/1606.03126 (2016).
  * Nallapati et al. (2017a) Ramesh Nallapati, Igor Melnyk, Abhishek Kumar, and Bowen Zhou. 2017a.  Sengen: Sentence generating neural variational topic model.  _arXiv preprint arXiv:1708.00308_ (2017).
  * Nallapati et al. (2017b) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017b.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In _AAAI_.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016\.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_ (2016).
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018\.  Ranking Sentences for Extractive Summarization with Reinforcement Learning. In _NAACL_. 1747–1759.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018\.  A Deep Reinforced Model for Abstractive Summarization. In _ICLR_.
  * Pavez et al. (2018) Juan Pavez, Hector Allende, and Hector Allende-Cid. 2018\.  Working Memory Networks: Augmenting Memory Networks with a Relational Reasoning Module. In _ACL_.
  * Ren et al. (2018) Pengjie Ren, Zhumin Chen, Z. Ren, Furu Wei, L. Nie, J. Ma, and M. Rijke. 2018\.  Sentence Relations for Extractive Summarization with Deep Neural Networks.  _TOIS_ 36 (2018), 1 – 32.
  * Ren et al. (2013) Zhaochun Ren, Shangsong Liang, Edgar Meij, and Maarten de Rijke. 2013.  Personalized time-aware tweets summarization. In _SIGIR_. ACM, 513–522.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015\.  A neural attention model for abstractive sentence summarization.  _arXiv preprint arXiv:1509.00685_ (2015).
  * Schluter (2017) Natalie Schluter. 2017\.  The limits of automatic summarisation according to ROUGE. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_. ACL, 41–45.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. ACL, 1073–1083.
  * Shen et al. (2019) Xiaoyu Shen, Yang Zhao, Hui Su, and Dietrich Klakow. 2019\.  Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3753–3764.
  * Steen and Markert (2019) Julius Steen and Katja Markert. 2019.  Abstractive Timeline Summarization. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_. 21–31.
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015.  End-To-End Memory Networks. In _NIPS_.
  * Sun et al. (2018) Min Sun, Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, and Jing Tang. 2018\.  A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss. In _ACL_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014\.  Sequence to Sequence Learning with Neural Networks. In _NIPS_.
  * Tao et al. (2018a) Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, and Rui Yan. 2018a.  Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism. In _IJCAI_. 4418–4424.
  * Tao et al. (2018b) Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018b.  RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems. In _AAAI_.
  * Tran et al. (2013) G. Tran, Tuan Tran, N. Tran, M. Alrifai, and Nattiya Kanhabua. 2013.  Leveraging Learning To Rank in an Optimization Framework for Timeline Summarization.
  * Wang et al. (2019b) Kai Wang, Xiaojun Quan, and Rui Wang. 2019b.  BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization. In _ACL_.
  * Wang et al. (2018b) Qinyong Wang, Hongzhi Yin, Zhiting Hu, Defu Lian, Hao Wang, and Zi Huang. 2018b.  Neural Memory Streaming Recommender Networks with Adversarial Training. In _KDD_.
  * Wang et al. (2019a) Wenbo Wang, Yang Gao, Heyan Huang, and Yuxiang Zhou. 2019a.  Concept Pointer Network for Abstractive Summarization. In _EMNLP_.
  * Wang et al. (2018a) Wenjie Wang, Minlie Huang, Xin-Shun Xu, Fumin Shen, and Liqiang Nie. 2018a.  Chat More: Deepening and Widening the Chatting Topic via A Deep Model. In _SIGIR_. ACM.
  * Wu et al. (2019) Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019\.  Global-to-local Memory Pointer Networks for Task-Oriented Dialogue. In _ICLR_.
  * Xiong et al. (2016) Caiming Xiong, Stephen Merity, and Richard Socher. 2016\.  Dynamic Memory Networks for Visual and Textual Question Answering.  _ArXiv_ abs/1603.01417 (2016).
  * Xu et al. (2020) Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020\.  Self-Attention Guided Copy Mechanism for Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 1355–1362.
  * Yan et al. (2011a) Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. 2011a.  Timeline generation through evolutionary trans-temporal summarization. In _EMNLP_. ACL, 433–443.
  * Yan et al. (2019) Rui Yan, Ran Le, Yang Song, Tao Zhang, Xiangliang Zhang, and Dongyan Zhao. 2019\.  Interview choice reveals your preference on the market: To improve job-resume matching through profiling memories. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 914–922.
  * Yan et al. (2012) Rui Yan, Xiaojun Wan, Mirella Lapata, Wayne Xin Zhao, Pu-Jen Cheng, and Xiaoming Li. 2012\.  Visualizing timelines: Evolutionary summarization via iterative reinforcement between text and image streams. In _CIKM_. ACM, 275–284.
  * Yan et al. (2011b) Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 2011b.  Evolutionary timeline summarization: a balanced optimization framework via iterative substitution. In _SIGIR_. ACM, 745–754.
  * Yan and Wan (2015) S. Yan and Xiaojun Wan. 2015.  Deep Dependency Substructure-Based Learning for Multidocument Summarization.  _TOIS_ 34 (2015), 3:1–3:24.
  * Yao et al. (2017) Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, and Rui Yan. 2017.  Towards Implicit Content-Introducing for Generative Short-Text Conversation Systems. In _EMNLP_.
  * Yasunaga et al. (2017) Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017.  Graph-based neural multi-document summarization.  _arXiv preprint arXiv:1706.06681_ (2017).
  * Zhang et al. (2020b) Hainan Zhang, Yanyan Lan, Liang Pang, Hongshen Chen, Zhuoye Ding, and Dawei Yin. 2020b.  Modeling Topical Relevance for Multi-Turn Dialogue Generation. In _IJCAI_.
  * Zhang et al. (2020a) Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2020a.  Structure Learning for Headline Generation.. In _AAAI_. 9555–9562.
  * Zhao et al. (2020) Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020\.  Low-Resource Knowledge-Grounded Dialogue Generation. In _ICLR_.
  * Zhao et al. (2013) Xin Wayne Zhao, Yanwei Guo, Rui Yan, Yulan He, and Xiaoming Li. 2013.  Timeline generation with social attention. In _SIGIR_. ACM, 1061–1064.
  * Zhong et al. (2020) Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020\.  Extractive Summarization as Text Matching.  _arXiv preprint arXiv:2004.08795_ (2020).
  * Zhou et al. (2019) Xiao Zhou, Cecilia Mascolo, and Zhongxiang Zhao. 2019\.  Topic-Enhanced Memory Networks for Personalised Point-of-Interest Recommendation. In _KDD_.
  * Zhu et al. (2020) Junnan Zhu, Y. Zhou, Jiajun Zhang, and Chengqing Zong. 2020\.  Attend, Translate and Summarize: An Efficient Method for Neural Cross-Lingual Summarization. In _ACL_.