# CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and
Summarization

Hossein Rajaby Faghihi1 , Bashar Alhafni2∗, Ke Zhang3, Shihao Ran3,
Joel Tetreault3, Alejandro Jaimes3
1Michigan State University, 2New York University Abu Dhabi,
3Dataminr, Inc.
rajabyfa<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done as Research Interns at
Dataminr, Inc.

###### Abstract

Social media has increasingly played a key role in emergency response: first
responders can use public posts to better react to ongoing crisis events and
deploy the necessary resources where they are most needed. Timeline extraction
and abstractive summarization are critical technical tasks to leverage large
numbers of social media posts about events. Unfortunately, there are few
datasets for benchmarking technical approaches for those tasks. This paper
presents CrisisLTLSum, the largest dataset of local crisis event timelines
available to date. CrisisLTLSum contains 1,000 crisis event timelines across
four domains: wildfires, local fires, traffic, and storms. We built
CrisisLTLSum using a semi-automated cluster-then-refine approach to collect
data from the public Twitter stream. Our initial experiments indicate a
significant gap between the performance of strong baselines compared to the
human performance on both tasks. Our dataset, code, and models are publicly
available.111 https://github.com/CrisisLTLSum/CrisisTimelines

##  1 Introduction

We present CrisisLTLSum, the first dataset on extraction and summarization of
local crisis event timelines from Twitter. An example of an annotated timeline
in CrisisLTLSum is shown in Figure 1. A timeline is a chronologically sorted
set of posts, where each brings in new information or updates about an ongoing
event (such as a fire, storm, or traffic incident). CrisisLTLSum supports two
complex downstream tasks: timeline extraction and timeline summarization. As
shown in Figure 1, the timeline extraction task is formalized as: given a seed
tweet as the initial mention of a crisis event, extract relevant tweets with
updates on the same event from the incoming noisy tweet stream. This task is
crucial for real-time event tracking. The timeline summarization task aims to
generate abstractive summaries of evolving events by aggregating important
details from temporal and incremental information.

Figure 1: This is a sample annotated timeline from CrisisLTLSum. The noisy
timeline is the set of tweets sorted chronologically. The first tweet is the
seed of the event. ✓ means that the tweet is annotated to be part of the
timeline and ✗ indicates that the tweet is excluded. The reason for the
exclusion is written under the mark.

CrisisLTLSum can facilitate research in two directions: 1) NLP for Social Good
(crisis domain), and 2) natural language inference and generation, i.e.,
timeline extraction and summarization tasks. Here, we discuss the importance
and the differences of CrisisLTLSum compared to previous work for both of
these aspects. Towards the first direction, the extraction of real-time
crisis-relevant information from microblogs Zhang and Eick (2019); Mamo et al.
(2021) plays a vital role in providing time-sensitive information to help
first responders understand ongoing situations and plan relief efforts
accordingly Sarter and Woods (1991). CrisisLTLSum goes beyond the task of
categorizing each single crisis-relevant post independently Imran et al.
(2013); Olteanu et al. (2014); Imran et al. (2016); Alam et al. (2018);
Wiegmann et al. (2020); Alam et al. (2021a, b) and enables a more challenging
task for extracting new updates of an ongoing crisis event from incoming posts
and summarizing them with respect to the important event details. This can
help provide time-sensitive updates while avoiding missing critical
information in the bulk of the posts in microblogs due to the high volume of
redundant and noisy information Alam et al. (2021a). To the best of our
knowledge, this is the first annotated dataset for such an extraction task,
while this problem has been tackled before in unsupervised settings Zhang et
al. (2018).

Moreover, we focus on the extraction of local crisis events. The term “local”
indicates that an event is bound to an exact location, such as a building, a
street, or a county, and usually lasts for a short period. Building a corpus
of local crisis events is particularly useful for first responders but also
challenging because the timelines of these events are often not captured in
existing knowledge sources. This means one has to design mechanisms for
automatically detecting and tracking events directly from the Twitter stream,
which is especially hard for existing clustering methods Guille and Favre
(2015); Asgari-Chenaghlu et al. (2021) given the low number of available
tweets for each local event.

For the second point, CrisisLTLSum enables NLP research on the complex tasks
of timeline extraction and abstractive summarization. These tasks are
particularly challenging in the context of social media. First, the process of
identifying and extracting relevant updates for a specific event has to
contend with the large volume of noise Alam et al. (2021a) and informal tone
Rudra et al. (2018) compared to other domains such as news. Additionally,
summarizing an on-going event helps toward a quick and better understanding of
its progress. This requires a good level of abstraction with important details
covered and properly presented (e.g., the temporal order of event evolution).
CrisisLTLSum is the first dataset to provide human-written timeline summaries
to support research in this direction.

CrisisLTLSum is developed through a two-step semi-automated process to create
1,000 local crisis timelines from the public Twitter stream. To our best
knowledge, this is the first timeline dataset focusing on “local” crisis
events with the largest number of unique events. The contributions of this
paper are as follows:

  * •

We propose CrisisLTLSum, which is the largest dataset over local crisis event
timelines. Notably, this is the first benchmark for abstractive timeline
summarization in the crisis domain or on Twitter.

  * •

We develop strong baselines for both tasks, and our experiments show a
considerable gap between these models and human performance, indicating the
importance of this dataset for enabling future research on extracting
timelines of crisis event updates and summarizing them.

##  2 Related Work

Our work in this paper is related to two main directions of crisis domain
datasets for NLP and timeline summarization.

Crisis Datasets for NLP: Prior research has investigated generating datasets
from online social media (e.g., Twitter) on large scale crisis events, while
providing labels for event categories Wiegmann et al. (2020); Imran et al.
(2013), humanitarian types and sub-types Olteanu et al. (2014); Imran et al.
(2016); Alam et al. (2018); Wiegmann et al. (2020); Arachie et al. (2020);
Alam et al. (2021a, b), actionable information McCreadie et al. (2019), or
witness levels Zahra et al. (2020) of each crisis related post. While existing
datasets on crisis event timelines Binh Tran et al. (2013); Tran et al.
(2015); Pasquali et al. (2021) are limited to a small set of large-scale
events, CrisisLTLSum covers a thousand timelines compared to only tens of
events covered by each of the existing datasets. Additionally, we further go
beyond the simple tweet categorization by enabling the extraction of
information that include updates over the events’ progress.

Timeline Summarization: Timeline summarization (TLS) was firstly proposed in
Allan et al. (2001), which extracts a single sentence from the news stream of
an event topic. In general, the TLS task aims to summarize the target’s
evolution (e.g., a topic or an entity) in a timeline Martschat and Markert
(2018); Ghalandari and Ifrim (2020). Existing approaches of TLS are mainly
based on extractive methods, which are often grouped into several categories.
For instance, Update Summarization Dang et al. (2008); Li et al. (2009) aims
to update the previous summary given new information at a later time, while
Timeline Generation Yan et al. (2011); Tran et al. (2015); Martschat and
Markert (2018) aims to generate itemized summaries as the timeline, where each
item is extracted by finding important temporal points (e.g., spikes, changes
or clusters) and selecting representative sentences. Another category,
Temporal Summarization, was first proposed in the TREC shared task Aslam et
al. (2013) with follow-up work Kedzie et al. (2015), which targets extracting
sentences from a large volume of news streams and social media posts as
updates for large events. Temporal Summarization is close to the first task
(Timeline Extraction) proposed in CrisisLTLSum.

There have been a few recent works on abstractive timeline summarization
across different domains, e.g., biography Chen et al. (2019), narratives
Barros et al. (2019), and news headlines Steen and Markert (2019), where the
human-written summaries are directly collected from the web. The abstractive
summarization goal is to generate a set of sentences summarizing the context
of interest without taking the exact words or phrases from the original text
but rather by combining them and summarizing the important content. To our
best knowledge, CrisisLTLSum is the first to provide human-written summaries
for crisis event timelines collected from noisy social media stream. Recent
research Nguyen et al. (2018) has also investigated the summarization task
based on tweets in other domains which essentially do not reflect the
challenges in the summarization of an evolving event.

##  3 CrisisLTLSum Collection

This section presents our semi-automated approach to collect CrisisLTLSum. We
first extract clusters of tweets as noisy timelines and then refine them via
human annotation to get clean timelines that only include non-redundant,
informative, and relevant tweets.

###  3.1 Noisy Timeline Collection

Figure 2 shows the process for generating a set of noisy timelines starting
from the Twitter stream and followed by pre-processing and knowledge
enhancement steps, the online clustering method, and post-processing &
cleaning steps.

Figure 2:  The process of noisy timeline collection. The output of this step
are noisy clusters which are used to create the dataset.

##### Location, Time, and Keywords Filtering

We limit the incoming tweets to specific geographical areas, periods, and
domains of interest.

The location filtering relies on a list of location candidates created by
gathering cities, towns, and famous neighborhoods in a big area of interest. A
tweet is considered relevant to our area of interest if 1) the text mentions
one of the candidates, 2) the geo-tag matches the area of interest, or 3) the
user location matches the area of interest. To limit the tweets to a specified
crisis domain, we curate domain-specific keywords and only select tweets with
phrases matching one of the keywords. This approach is not comprehensive or
exhaustive but somewhat representative of each crisis domain. Improving this
method to be more encompassing is an area for future research. The
combinations of (area $a$, domain $d$, time $t$) are manually selected so that
the events of type $d$ at location $a$ are more frequent during time period
$t$. For instance, wildfire events are most likely to happen in California
from May through August, while the same type of event is more likely from
December to Match in Victoria (Australia). More details with examples of
curated keywords can be found in Appendix A.

##### Entity Extraction

This step aims to extract entity mentions from the tweet text and provide
additional information that can be used to help identify related tweets. We
use three different modules. First, we use a pre-trained neural model from
AllenNLP Gardner et al. (2018), trained on CoNLL03 Tjong Kim Sang and De
Meulder (2003), to extract entities with types of people, location, and
organization. Although this module extracts some important entities in the
text, it fails to extract uncommon entities or special mentions such as the
name of a wildfire. To address this, similar to prior research Zheng and
Kordjamshidi (2020), we further exploit the extractions from OpenIE Stanovsky
et al. (2018) and select the noun arguments with less than ten characters as
entities. Lastly, we add the tweet’s hashtags to the entity set. Since
location mentions are crucial in extracting local events and existing models
have low performance detecting them from noisy tweets text, we further
developed a BERT-based NER model tuned on Twitter data to detect location
mentions.

##### Location Augmentation

We use OpenStreetMap API to map location mentions to physical
addresses.222https://www.openstreetmap.org/ This step provides complementary
information about each location while reducing the noise introduced by the
entity extraction module through removing location mentions that are wrongly
detected or are not located in the area of interest. This is especially
important since our focus is on local events happening at specific locations.

##### Online Clustering

This step aims to mimic the real-life scenario where tweets are sequentially
fed into a clustering algorithm Wang et al. (2015). We further choose this
method since this is a lot faster than the retro-respective (all data
available at the same time) clustering methods for a large pool of input data.
Here, the clustering objective is to group tweets related to the same local
event, such as a “fire in building A” or a “wildfire in a specific area”. The
online clustering method utilizes a custom similarity metric that combines the
similarity of the entities, the closeness of locations in the real world, and
the existence of shared hashtags.

Algorithm 1 Find Similarity of tweet $t_{i}$ and $t_{j}$

$similarity=0$

if $t_{1}.hashtags\cap t_{2}.hashtags\neq\emptyset$ then

$similarity=similarity+s_{hashtag}$

end if

$dist_{min}=smallest\\_distance(t_{1},t_{2})$

if $dist_{min}\geq max_{dist}$ then

return 0

else if $dist_{min}\leq min_{dist}$ then

$similarity=similarity+s_{dist}$

end if

$top\\_pairs=$ find_matching_entities$(t_{1},t_{2})$

$norm\\_factor=$

$\min(len(t_{1}$.entities$),len(t_{2}$.entities$),top\\_pairs)$

$s_{entity}=\sum top\\_pairs.similarity/norm\\_factor$

$similarity=similarity+s_{entity}$

return $similarity$

Algorithm 1 shows the similarity computation between two tweets. The
smallest_distance computes the minimum physical distance between location
mentions given their augmented real-world location (the output of the location
augmentation step). As the distance between higher-level location mentions
such as state/city/country is always zero, we simply ignore those location
types. The find_matching_entities function follows the ideas in Faghihi et al.
(2020) on creating a unique matching matrix, which we use for extracting the
top matching pairs of entities from tweets. Here, each entity can only be
paired once with the highest matching-score entity from the other tweet. The
$min_{dist}$, $max_{dist}$, $s_{hashtag}$, and $s_{dist}$ are hyper-parameters
of the clustering algorithm. We have only used heuristics and a small set of
executions to tune these hyper-parameters.

The pre-processed set of tweets is passed to the online clustering method, one
tweet at a time. For each new tweet, similarity scores are computed between
the new tweet and all cluster heads. The new tweet is added to the highest
matching-score cluster where the similarity score is higher than
$sim_{threshold}$ and the time elapsed between the new tweet and the last
update of the cluster is less than $time_{threshold}$. If the previous
criteria are met for none of the clusters, a new cluster is created based on
the new tweet. During this process, we remove inactive clusters whose last
update was at least $expiration_{threshold}$ minutes ago and have less than
$tweet_{threshold}$ number of tweets available. A cluster head is always the
tweet with the most entity mentions; In case of a tie, the more recent tweet
becomes the head of the cluster. The hyper-parameters of this method is noted
in Appendix A.

###### Cluster Post-Processing

We apply three post-processing steps to improve the quality of the generated
clusters. First, we manually merge pairs of clusters with a cluster head
similarity higher than a threshold $head_{min}$. This step compensates for
some of the errors from missing entities in the pre-processing step, which
affects the intermediate similarity scores in the clustering algorithm.
Second, we use a simple fuzzy sequence-matching technique to remove identical
or similar tweets inside each cluster. Third, we train a BERT-based Devlin et
al. (2019) binary classifier to detect informative content, which can be used
to prune out the noisy tweets that do not include crisis-relevant information.
This classifier is trained on the available labeled data Alam et al. (2021a)
on tweets’ informativeness. Since most of the available tweets in Alam et al.
(2021a) are specific to storm and wildfire domains and there are no
representative subsets for our other domains of interest (traffic, local
fire), we only apply this step to the clusters that are generated for those
categories. These post-processing steps aim not to prune out all the noisy
information but rather to provide a better starting point for our next steps.

###  3.2 CrisisLTLSum Human Annotation

Taking the noisy timelines generated from the previous step, we leverage human
annotations to refine and generate clean timelines and summarize them. We,
authors of this work, manually selected 1,000 clusters that contain enough
tweets describing how a crisis event evolves, while specifying the “seed
tweet” (i.e. the first observed post that describes the ongoing event) of each
timeline. The detailed process is presented in Appendix B. The selected
clusters cover events mainly from four crisis domains, including wildfire,
local fire, storm, and traffic. More data statistics are shared in Section 4.

##### Procedure

We use the Amazon Mechanical Turk (MTurk) platform to label and refine the
noisy clusters to generate a clean timeline and collect the summaries. We
split the annotation into multiple batches of Human Intelligence Tasks (HITs),
where each batch contains timelines from the same domain. Each HIT contains
three noisy timelines, and we collect annotations from 3 different workers on
each. The workers are given the seed tweet and the subsequent tweets sorted by
time, and they were asked to read the tweet one by one and answer i) whether
the tweet should be part of the timeline, and ii) what is the reason if not.

A tweet is labeled as part of the timeline only if it satisfies all the
following three conditions:

  * •

relevant: talks about the same event indicated in the seed tweet

  * •

informative: provides facts about the event but not only contains personal
points of view

  * •

not repetitive: brings in new information about the ongoing event

After reviewing all the tweets, the worker is finally asked to write a concise
summary to describe how the event progresses over time. Detailed instructions
and annotation workflows are presented as Figures 8-14 in Appendix E.

##### Annotation Workflow & Quality Control

Following prior quality control practices Briakou et al. (2021), we use
multiple quality control (QC) steps to ensure the recruitment of high-quality
annotators. First, we use location restriction (QC1) to limit the pool of
workers to countries where native English speakers are most likely to be
found. Next, we recruit annotators who pass our qualification test (QC2),
where we ask them to annotate 3 timelines. We run several small pilot tasks,
each with a replication factor of nine. We check annotators’ performance on
timeline extraction task against experts’ labels and have experts manually
review (QC3) annotators’ summary quality. Only workers passing all the quality
control steps contribute to the final task. During the final task, we perform
regular quality checks (QC4), and only use workers who consistently perform
well.

##### Compensation

We compensate the workers at a rate of $3 per HIT for the task. Each batch of
tasks is followed by a one-time bonus that makes the final rate over $10 per
hour.

##  4 CrisisLTLSum Statistics & Analysis

In this section, we cover comprehensive statistics and analysis of
CrisisLTLSum to further elaborate on the statistical characteristics of our
dataset.

Domain | Timelines | Tweets | ✓ | ✗
---|---|---|---|---
Wildfire | 423 | 4,829 | 1,961 | 2,868
Traffic | 287 | 2,340 | 831 | 1,509
Fire | 155 | 1,469 | 640 | 829
Storm | 109 | 1,767 | 789 | 978
Other | 26 | 205 | 82 | 123
| 1,000 | 10,610 | 4,303 | 6,307
Table 1: Data statistics across different crisis domains in terms of the
number of timelines and tweets. ✓ indicates tweets that are part of the
timeline and ✗ indicates tweets that are not part of the timeline.

###  4.1 Dataset Statistics

Out of these 1,000 annotated timelines (10,610 tweets) in CrisisLTLSum, 423
(42%) are about wildfire, 287 (29%) are about traffic, 155 (15%) are about
local fire, 109 (11%) are about storm, and 26 (3%) are about other types of
events (e.g., building collapse). To make the ground-truth on whether each
tweet is part of the timeline or not, we take the majority label among the
three workers. 4,303 (41%) tweets are labeled as part of the timeline, whereas
6,307 (59%) are not. Out of all timelines, 110 (11%) only include tweets that
are part of the timeline, whereas 68 (7%) did not include any. Table 1
presents the statistics across different event crisis domains.

###  4.2 Dataset Analysis

##### Timelines by Lengths

Table 2 presents the aggregated length (i.e., number of tweets) distribution
of the timelines. The majority of the timelines (447 or 45%) have five tweets
or less. This is observed across all crisis domains except for storm events,
where the majority of the timelines (43 out of 109) are 6 to 12 tweets in
length. It is worth noting that our dataset includes long timelines of 26 or
more tweets, constituting 9% (94 timelines) across all domains. Figure 3
presents the average percentage of part-of-timeline tweets based on the
aggregated length distributions of the timelines. We notice an interesting
trend across the domains: the longer the timeline, the lower the average
percentage of tweets to be part of the timeline.

##### Annotation Quality

To measure the agreement rate between workers on the timeline extraction task,
we consider the annotations of two out of three workers who agree most per
timeline. The average timeline-level agreement between those two workers is
90.06%. We also explore a more profound analysis by comparing the workers’
annotations on 20% of the timelines against the annotations of experts. To do
so, we evaluate the majority label provided by the three workers against the
label provided by the experts. The average timeline-level agreement rate of
this analysis is 91.77%.

###  4.3 Dataset Splits

To aid reproducibility when using our dataset for various research
experiments, we divided our dataset into: training (Train: 70% or 706
timelines), development (Dev 10% or 86 timelines), and testing (Test 20% or
208 timelines) splits. The splits are created via stratified sampling based on
the event crisis domains and the length of timelines (i.e., number of tweets)
as shown in Table 2. Detailed statistics are available in Appendix D.

Domain | Aggregated Timelines Lengths
---|---
[1 – 5] | [6–12] | [13–25] | [26 –]
Wildfire | 175 | 140 | 59 | 49
Traffic | 158 | 95 | 20 | 14
Fire | 76 | 45 | 22 | 12
Storm | 21 | 43 | 27 | 18
Other | 17 | 8 | 0 | 1
| 447 | 331 | 128 | 94
Table 2: Dataset statistics based on the aggregated timelines lengths (i.e.,
number of tweets) across different crisis domains. Figure 3: Average % of
tweets that are part of the timelines based on the aggregated timelines
lengths across different crisis domains.

##  5 Experiments

Here, we first formalize the definitions of the downstream tasks. We then
propose a set of existing naive and advanced models to serve as a baseline for
these tasks. Next, we analyze the performance of the baselines on each task
and provide error analysis for the best performing baseline to indicate the
remaining major challenges of this dataset for future research.

###  5.1 Task Definitions

Given a crisis event timeline consisting of $n$ tweets
$T=[t_{0},t_{1},...,t_{n}]$, where $t_{0}$ is the seed tweet, i.e. the first
observed post that describes the ongoing event, we define two tasks: Timeline
Extraction and Timeline Summarization.

Timeline Extraction:  Given an initial seed tweet $t_{0}$, and following
chronologically sorted tweets $t_{i},i=1,...,n$, the goal of the timeline
extraction task is to determine whether tweet $t_{i}$ is part of the timeline,
i.e., related to the same event and adding new information compared to all
prior tweets $[t_{0},...,t_{i-1}]$. Up to time $n$, a clean timeline
$T_{extracted}=[t_{0},t_{e}^{1},t_{e}^{2},..,t_{e}^{m}],m\leq n$ is extracted
to describe how the event progresses over time.

Timeline Summarization:  Similar to previous work by Chen et al. (2019), this
task aims to generate a summary $\hat{Y}=\\{\hat{w}_{1},...,\hat{w}_{k}\\}$
with sequence of words $\hat{w}_{i}$ to concisely describe the crisis event
and its evolution given the output from the timeline extraction task
$T_{extracted}$. In particular, it tries to optimize the parameters that
maximize the probability $P(Y/T_{extracted})$ given ground-truth summaries
$Y=\\{w_{1},...,w_{k}\\}$.

###  5.2 Modeling

####  5.2.1 Timeline Extraction

##### Naive Baseline

We employ a simple and naive baseline that assigns the label of the majority
class observed in Train to detect whether a tweet is part of the timeline or
not.

##### Sequence Classification

We leverage pretrained language models to build sentence-level classifiers. We
construct a list of tweet sequences $S$ by concatenating all the tweets that
are part of the timeline from $t_{1}$ to $t_{i}$:
$S=[t_{1},t_{1}+t_{2},...,t_{1}+...+t_{n}]$. Therefore, each timeline of $n$
tweets would yield $n$ sequences. For our first sentence-level classification
model, we fine-tune BERT Devlin et al. (2019) on the tweet sequences where
every training example would have the form of “$t_{0}$ [SEP] $s_{i}$”; where
$s_{i}\in S$. We fine-tune this model using Hugging Face’s Transformers Wolf
et al. (2020).

##### Sequence Labeling

Given the sequential nature of timelines, we treat the timeline extraction
task as a sequence labeling problem. We create a model by adding a GRU Cho et
al. (2014) on top of BERT. Given a timeline $T=[t_{0},t_{1},t_{2},..,t_{n}]$,
we feed every tweet $t_{i}$ to BERT and get its contextual representation
corresponding to the [CLS] token. We then concatenate the representations of
all the tweets and feed them to the GRU to predict if each tweet is part of
the timeline or not.333It is important to note that the loss corresponding to
the seed tweet $t_{0}$ is masked out during training.

####  5.2.2 Timeline Summarization

##### Naive Baselines

We define three simple models as the first set of timeline summarization
baselines: 1) first-tweet: uses the first tweet of the timeline as the output
summary; 2) last-tweet: uses the last tweet of the timeline as the output
summary; and 3) random-tweet: uses a random tweet from the timeline as the
output summary.

##### Seq2Seq Models

We further benchmark two pretrained sequence-to-sequence (Seq2Seq) models:
BART Lewis et al. (2020) and DistillBART Shleifer and Rush (2020). We chose
these two models as they achieve strong results on various summarization
datasets. For BART, we train it from scratch, and for DistillBART, we use a
version that has been fine-tuned on both XSum Narayan et al. (2018) and CNN/DM
Hermann et al. (2015) datasets. We use the fine-tuned DistillBART model in two
settings: 1) further fine-tuning on CrisisLTLSum; 2) zero-shot setting. We use
HuggingFace’s Transformers to fine-tune both of these models.

The input to the Seq2Seq models is the concatenation of all tweets that are
part of the timeline. Since each timeline in our dataset is annotated by three
workers (§3.2), it includes three summaries. To adapt the timeline
summarization task to this setting, we pick the two summaries written by the
two workers who agree the most based on the timeline extraction labels they
assign to the tweets in each timeline. This reduces the variance between the
summaries regarding their coverage of the crisis event described in the tweets
that are part of the timeline. During fine-tuning, we double the examples in
Train by repeating each timeline twice, once for each summary. We describe all
training settings and hyperparameters in Appendix C.

##  6 Results

###  6.1 Timeline Extraction

Table 3 presents timeline extraction results on the Dev set. For evaluation,
we use the average timeline-level accuracy. The naive majority class baseline
marks every tweet as not part of the timeline and achieves 48.57 average
timeline accuracy. This is expected since 10 timelines in the Dev set only
contain tweets that are part of the timeline (§4.1). The BERT-based sequence
classification model achieves 74.64 average timeline accuracy, beating the
BERT-GRU model, which achieves 65.86. Although the BERT-based sequence
classification model has a limitation when extracting long timelines due to
its limited size of 512 positional embeddings, it performs better than the
BERT-GRU sequence labeling model. We attribute this to: 1) the careful
preprocessing we did when constructing the sequences used to train the BERT-
based sequence classification model, and 2) the limited data size, which might
not enable the BERT-GRU model to fully capture the sequential relationship
across the tweets in the timelines.

For timeline extraction results on the Test set, we compare both the human-
level performance and the best model’s performance against experts’
annotations.444It is worth noting that this is the same set that was used to
estimate the annotation quality (§4.2) To get the human-level performance on
the Test set, we average the performance of the two workers who agree most
across all timelines in Test. The best timeline extraction model achieves
73.51 average timeline accuracy, whereas the human-level performance is 88.98.
This highlights the difficulty of the timeline extraction task and indicates
that more involved models are needed to close the gap between model
performance and human-level performance.

###  6.2 Timeline Summarization

We evaluate all timeline summarization models with ROUGE Lin and Och (2004).
The summarization output of each model is evaluated in a multi-reference
setting against the two summaries written by the two workers who agree the
most based on the timeline extraction labels. We use SacreROUGE Deutsch and
Roth (2020) to compute multi-reference ROUGE scores.

Table 4 presents the F1 scores of ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-L (RL)
of the timeline summarization models on the Dev set. The three naive baselines
(first tweet, last tweet, and random tweet) achieve comparable performances,
with R1 being the highest compared to R2 and RL. This is expected since the
tweets in the timeline and the reference summaries (also tweets) all describe
the same event, which leads to a considerable unigram overlap (i.e., R1).

For the Seq2Seq pretrained models, we present results in two settings: 1)
using the oracle timeline; and 2) using the extracted timeline from the best
timeline extraction model. In the oracle setting, we use the gold labels in
the Dev set to identify tweets that are part of each timeline to the
summarization models. The results of this oracle experiment estimate an upper
bound for the summarization models used in this task. As shown in Table 4, the
fine-tuned BART model achieves the best performance in terms of R1, R2 and RL.
When we apply the best timeline extraction models to identify the tweets that
are part of the timeline, we observe consistent conclusions. On Test set, the
best summarization model (i.e., BART) achieves 47.05, 25.40, and 35.90 in R1,
R2, and RL, respectively.

| Accuracy
---|---
Majority Class | 48.57
BERT | 74.64
BERT-GRU | 65.86
Table 3: Results of timeline extraction models on Dev.

| R1 | R2 | RL
---|---|---|---
First tweet | w/o TE | 32.15 | 14.35 | 24.09
Last tweet | w/o TE | 28.69 | 12.03 | 21.65
Random tweet | w/o TE | 30.56 | 13.61 | 23.24
BART | Oracle | 47.72 | 26.16 | 36.90
TE | 45.32 | 24.61 | 34.68
DistillBART | Oracle | 44.87 | 24.37 | 32.00
(Zero-shot) | TE | 42.97 | 21.82 | 29.49
DistillBART | Oracle | 47.50 | 25.11 | 34.64
TE | 45.04 | 22.61 | 32.51

Table 4: Results of different timeline summarization models after timeline
extraction and in oracle settings on Dev.

| [All] | [6 –]
---|---|---
| Acc | Coherence | Cov | Overall | Acc | Coherence | Cov | Overall
Human | 4.75 | 4.85 | 4.14 |  4.12⋆ |  4.73⋆ | 4.83 |  3.97⋆ |  3.98⋆
BART | 4.71 | 4.86 | 3.91 | 3.93 | 4.59 | 4.81 | 3.43 | 3.50
DistillBART | 4.33 | 4.68 | 4.17 | 3.90 | 4.56 | 4.70 | 3.82 | 3.74
DistillBART (Zero-shot) | 4.60 | 4.60 | 4.13 | 3.92 | 4.67 | 4.83 | 3.75 | 3.7
Random Tweet | 4.72 | 4.71 | 3.40 | 3.40 | 4.53 | 4.64 | 2.72 | 2.79
Random Summary | 1.16 | 4.55 | 1.16 | 1.17 | 1.24 | 4.45 | 1.25 | 1.25

Table 5: Results of human evaluation on human and model generated summaries.
[All] means all timelines in the Testand [6 –] means ones with length $\geq
6$. ⋆ denotes human summary is significantly (p < 0.05) better or worse than
the best performing model based on one-sided Mann–Whitney U test Mann and
Whitney (1947). Bold numbers denote best systems.

##### Human Evaluation

We further conducted a human evaluation to better assess the quality of the
summaries. We take the whole Test set and we evaluate eight summaries per
timeline: 1) the three human-written summaries; 2) three model-generated
summaries using the three models we develop for timeline summarization in the
oracle setting; 3) two summaries from random systems: a random tweet that is
part of the timeline, and a summary from a randomly selected timeline that
belongs to a different crisis domain. The random systems add naive baselines
and also serve as additional check on the annotation quality. Following a
similar process in 3.2, we recruit a group of workers from MTurk with the same
Location restriction (QC1), who can pass our pilot study as a qualification
test. Each summary was evaluated by five different workers on a scale from 1
to 5 across four axes: Coherence, Accuracy, Coverage, and Overall quality, as
was done by Lai et al. (2022). Detailed instructions and annotation workflows
are presented as Figures 15–21 in Appendix E.

Table 5 presents the results. Looking at the results over all the timelines,
the human-written summaries are significantly better than the ones generated
by models, in terms of overall quality with an average rating of $4.12$. The
human-written summaries are also better in terms of accuracy, however, this
was not statistically significant compared to the average accuracy ratings
assigned to the models generated summaries. The models generated summaries
have higher average ratings in terms of coherence and coverage, but this is
not statistically significant compared to the human-written summaries.
Notably, the gap of the ratings becomes larger for longer timelines (i.e.,
timelines with 6 or more tweets). The human-written summaries are
significantly better in terms of accuracy, coverage, and overall. In terms of
coherence, they are on par with the models generated summaries. This
highlights the shortcomings of the summarization models when it comes to long
timelines.

###  6.3 Error Analysis

We conduct an error analysis over the outputs of the best models for timeline
extraction and summarization on Dev. For timeline extraction, 66 timelines
(77%) have at least one error, in which 272 out of 958 tweets (28%) are wrong.

| Extraction | Summarization
---|---|---
Length | Accuracy | R1 | R2 | RL
[1–5] | 76.97 | 49.43 | 29.48 | 39.25
[6–12] | 74.24 | 43.92 | 23.19 | 33.62
[13–25] | 73.59 | 43.71 | 22.93 | 32.65
[26–] | 67.37 | 35.10 | 11.87 | 21.73
Table 6: Best timeline extraction and summarization models performances on Dev
based on timelines’ lengths.

Table 6 presents timeline extraction and summarization results of the best
models based on different timeline lengths over the Dev set. We observe that
the model performance decreases with increasing timeline length for both
timeline extraction and summarization tasks, with significant drops in
accuracy and ROUGE scores when timelines get longer. Moreover, we inspected
some of the generated summaries manually and we noticed that most of the
summarization errors were due to hallucinations or to copying specific
sentences that are present in the timeline without covering all the important
event details described in the timeline. For instance, for the timeline in
Figure 1, the best baseline summarization model (i.e., BART) generated the
following summary: “A large fire burning in northeast Fresno near Woodward
Lake sent plumes of smoke into the air above the city. Officials say the fire
started as a commercial fire”. The two sentences in the generated summary are
copied verbatim from the tweets in the timeline. This highlights the need for
better models to capture the important details mentioned in the timeline.

##  7 Conclusion

We presented CrisisLTLSum, the first dataset on local crisis timelines
extracted from Twitter and the first to provide human-written summaries on
information extracted from Twitter. We showed that CrisisLTLSum supports two
downstream tasks: Timeline Extraction and Timeline Summarization. Our
experiments with SOTA baselines indicate that both of these tasks are
challenging and encourage future research. Our dataset further provides a
resource for developing methods on utilizing micro-blogs toward aiding first-
responders in evaluating ongoing crisis events. In future, we plan to explore
models that can solve both tasks in a joint setting by extracting new
information from each update point and then summarizing those. This dataset
can also be expanded by additional annotation to enable abstractive entity-
based understanding of the event flow Mishra et al. (2018); Faghihi and
Kordjamshidi (2021).

## Acknowledgements

We thank Aoife Cahill for providing detailed feedback on this paper, Alison
Smith-Renner and Wenjuan Zhang for helpful discussions on MTurk annotations,
our colleagues at Dataminr, and the emnlp reviewers for their helpful and
constructive comments.

## Limitations

This study has some limitations on the dataset generation workflow. First, our
noisy timeline collection process is not a comprehensive and exhaustive
extraction to find all available information about a local crisis event. Here,
our focus is to provide a representative dataset rather performing a
comprehensive set containing all the local crisis events and their updates.
Second, the proposed noisy timeline collection pipeline is highly dependent to
the performance of the entity extraction modules, especially for the location
extraction, and the accuracy of the OpenStreetMap API to find the correct
physical address of each location mention. Accordingly, replicating the same
process for other language or based on other locations may be hard because of
such dependencies. Furthermore, the online clustering method used in this
paper has a set of hyperparameters which are tuned heuristically from a small
set of experiments. More comprehensive and large scale experiments on tuning
those parameters could potentially impact the quality of the generated
timelines. Generating similar results by following our noisy timeline
collection process is in general limited by the users’ access to the public
Twitter stream and the changes in the available posts (they may become
restricted or deleted).

## References

  * Alam et al. (2018) Firoj Alam, Ferda Ofli, and Muhammad Imran. 2018.  Crisismmd: Multimodal twitter datasets from natural disasters.  In _Twelfth international AAAI conference on web and social media_.
  * Alam et al. (2021a) Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Ofli. 2021a.  Humaid: Human-annotated disaster incidents data from twitter with deep learning benchmarks.  In _ICWSM_ , pages 933–942.
  * Alam et al. (2021b) Firoj Alam, Hassan Sajjad, Muhammad Imran, and Ferda Ofli. 2021b.  Crisisbench: Benchmarking crisis-related social media datasets for humanitarian information processing.  In _ICWSM_ , pages 923–932.
  * Allan et al. (2001) James Allan, Rahul Gupta, and Vikas Khandelwal. 2001.  Temporal summaries of new topics.  In _Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 10–18.
  * Arachie et al. (2020) Chidubem Arachie, Manas Gaur, Sam Anzaroot, William Groves, Ke Zhang, and Alejandro Jaimes. 2020.  Unsupervised detection of sub-events in large scale disasters.  In _Proceedings Of The AAAI conference on artificial intelligence_ , volume 34, pages 354–361.
  * Asgari-Chenaghlu et al. (2021) Meysam Asgari-Chenaghlu, Mohammad-Reza Feizi-Derakhshi, Leili Farzinvash, Mohammad-Ali Balafar, and Cina Motamed. 2021.  Topic detection and tracking techniques on twitter: a systematic review.  _Complexity_ , 2021.
  * Aslam et al. (2013) Javed A Aslam, Matthew Ekstrand-Abueg, Virgil Pavlu, Fernando Diaz, and Tetsuya Sakai. 2013.  Trec 2013 temporal summarization.  In _TREC_.
  * Barros et al. (2019) Cristina Barros, Elena Lloret, Estela Saquete, and Borja Navarro-Colorado. 2019\.  Natsum: Narrative abstractive summarization through cross-document timeline generation.  _Information Processing & Management_, 56(5):1775–1793.
  * Binh Tran et al. (2013) Giang Binh Tran, Mohammad Alrifai, and Dat Quoc Nguyen. 2013.  Predicting relevant news events for timeline summaries.  In _Proceedings of the 22nd International Conference on World Wide Web_ , pages 91–92.
  * Briakou et al. (2021) Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreault. 2021.  Olá, bonjour, salve! xformal: A benchmark for multilingual formality style transfer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3199–3216.
  * Chen et al. (2019) Xiuying Chen, Zhangming Chan, Shen Gao, Meng-Hsuan Yu, Dongyan Zhao, and Rui Yan. 2019.  Learning towards abstractive timeline summarization.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 4939–4945. International Joint Conferences on Artificial Intelligence Organization.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Dang et al. (2008) Hoa Trang Dang, Karolina Owczarzak, et al. 2008.  Overview of the tac 2008 update summarization task.  In _TAC_.
  * Deutsch and Roth (2020) Daniel Deutsch and Dan Roth. 2020.  SacreROUGE: An open-source library for using and developing summarization evaluation metrics.  In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_ , pages 120–125, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Faghihi and Kordjamshidi (2021) Hossein Rajaby Faghihi and Parisa Kordjamshidi. 2021.  Time-stamped language model: Teaching language models to understand the flow of events.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4560–4570.
  * Faghihi et al. (2020) Hossein Rajaby Faghihi, Roshanak Mirzaee, Sudarshan Paliwal, and Parisa Kordjamshidi. 2020.  Latent alignment of procedural concepts in multimodal recipes.  In _Proceedings of the First Workshop on Advances in Language and Vision Research_ , pages 26–31.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018.  AllenNLP: A deep semantic natural language processing platform.  In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_ , pages 1–6, Melbourne, Australia. Association for Computational Linguistics.
  * Ghalandari and Ifrim (2020) Demian Gholipour Ghalandari and Georgiana Ifrim. 2020.  Examining the state-of-the-art in news timeline summarization.  _arXiv preprint arXiv:2005.10107_.
  * Guille and Favre (2015) Adrien Guille and Cécile Favre. 2015.  Event detection, tracking, and visualization in twitter: a mention-anomaly-based approach.  _Social Network Analysis and Mining_ , 5(1):1–18.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _NIPS_ , pages 1693–1701.
  * Imran et al. (2013) Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, and Patrick Meier. 2013.  Practical extraction of disaster-relevant information from social media.  In _Proceedings of the 22nd international conference on world wide web_ , pages 1021–1024.
  * Imran et al. (2016) Muhammad Imran, Prasenjit Mitra, and Carlos Castillo. 2016.  Twitter as a lifeline: Human-annotated twitter corpora for nlp of crisis-related messages.  _arXiv preprint arXiv:1605.05894_.
  * Kedzie et al. (2015) Chris Kedzie, Kathleen McKeown, and Fernando Diaz. 2015.  Predicting salient updates for disaster summarization.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1608–1617.
  * Lai et al. (2022) Vivian Lai, Alison Smith-Renner, Ke Zhang, Ruijia Cheng, Wenjuan Zhang, Joel Tetreault, and Alejandro Jaimes-Larrarte. 2022.  An exploration of post-editing effectiveness in text summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 475–493, Seattle, United States. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2009) Liangda Li, Ke Zhou, Gui-Rong Xue, Hongyuan Zha, and Yong Yu. 2009.  Enhancing diversity, coverage and balance for summarization through structure learning.  In _Proceedings of the 18th international conference on World wide web_ , pages 71–80.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 605–612, Barcelona, Spain.
  * Mamo et al. (2021) Nicholas Mamo, Joel Azzopardi, and Colin Layfield. 2021.  Fine-grained topic detection and tracking on twitter.  In _KDIR_ , pages 79–86.
  * Mann and Whitney (1947) H. B. Mann and D. R. Whitney. 1947.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other.  _The Annals of Mathematical Statistics_ , 18(1):50 – 60.
  * Martschat and Markert (2018) Sebastian Martschat and Katja Markert. 2018.  A temporally sensitive submodularity framework for timeline summarization.  _arXiv preprint arXiv:1810.07949_.
  * McCreadie et al. (2019) Richard McCreadie, Cody Buntain, and Ian Soboroff. 2019.  Trec incident streams: Finding actionable information on social media.
  * Mishra et al. (2018) Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018\.  Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension.  _arXiv preprint arXiv:1805.06975_.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Nguyen et al. (2018) Minh-Tien Nguyen, Dac Viet Lai, Huy Tien Nguyen, and Minh Le Nguyen. 2018.  Tsix: a human-involved-creation dataset for tweet summarization.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Olteanu et al. (2014) Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2014.  Crisislex: A lexicon for collecting and filtering microblogged communications in crises.  In _Eighth international AAAI conference on weblogs and social media_.
  * Pasquali et al. (2021) Arian Pasquali, Ricardo Campos, Alexandre Ribeiro, Brenda Santana, Alípio Jorge, and Adam Jatowt. 2021.  Tls-covid19: A new annotated corpus for timeline summarization.  In _Advances in Information Retrieval_ , pages 497–512, Cham. Springer International Publishing.
  * Rudra et al. (2018) Koustav Rudra, Niloy Ganguly, Pawan Goyal, and Saptarshi Ghosh. 2018.  Extracting and summarizing situational information from the twitter social media during disasters.  _ACM Transactions on the Web (TWEB)_ , 12(3):1–35.
  * Sarter and Woods (1991) Nadine B Sarter and David D Woods. 1991.  Situation awareness: A critical but ill-defined phenomenon.  _The International Journal of Aviation Psychology_ , 1(1):45–57.
  * Shleifer and Rush (2020) Sam Shleifer and Alexander M. Rush. 2020.  Pre-trained summarization distillation.
  * Stanovsky et al. (2018) Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and I. Dagan. 2018.  Supervised open information extraction.  In _NAACL-HLT_.
  * Steen and Markert (2019) Julius Steen and Katja Markert. 2019.  Abstractive timeline summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 21–31, Hong Kong, China. Association for Computational Linguistics.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Tran et al. (2015) Giang Tran, Mohammad Alrifai, and Eelco Herder. 2015.  Timeline summarization from relevant headlines.  In _European Conference on Information Retrieval_ , pages 245–256. Springer.
  * Wang et al. (2015) Zhenhua Wang, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2015.  On summarization and timeline generation for evolutionary tweet streams.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(5):1301–1315.
  * Wiegmann et al. (2020) Matti Wiegmann, Jens Kersten, Friederike Klan, Martin Potthast, and Benno Stein. 2020.  Disaster tweet corpus 2020.  _Disaster Tweet Corpus 2020 [Data Set]_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yan et al. (2011) Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 2011.  Evolutionary timeline summarization: a balanced optimization framework via iterative substitution.  In _Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval_ , pages 745–754.
  * Zahra et al. (2020) Kiran Zahra, Muhammad Imran, and Frank O Ostermann. 2020.  Automatic identification of eyewitness messages on twitter during disasters.  _Information processing & management_, 57(1):102107.
  * Zhang et al. (2018) Chao Zhang, Dongming Lei, Quan Yuan, Honglei Zhuang, Lance Kaplan, Shaowen Wang, and Jiawei Han. 2018.  Geoburst+ effective and real-time local event detection in geo-tagged tweet streams.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 9(3):1–24.
  * Zhang and Eick (2019) Yongli Zhang and Christoph F Eick. 2019.  Tracking events in twitter by combining an lda-based approach and a density–contour clustering approach.  _International Journal of Semantic Computing_ , 13(01):87–110.
  * Zheng and Kordjamshidi (2020) Chen Zheng and Parisa Kordjamshidi. 2020.  Srlgrn: Semantic role labeling graph reasoning network.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8881–8891.

##  Appendix A Noisy Timeline Collection

In this section, we provide some additional information on the noisy timeline
collection process.

###  A.1 Location and Date Selection

We build CrisisLTLSum by limiting the search to four crisis domains: wildfire,
local fire, traffic, and storm. We collect CrisisLTLSum across seven states in
the USA and two states of Australia during 2020 and 2021. The selected states
in the USA are New York, Illinois, Massachusetts, California, Texas, and
Florida. From Australia, we have selected the states of New South Wales and
Victoria. These states are selected as they are some of the areas subject to
the most events falling into our crisis domains of interest. California,
Victoria, and New South Wales are mainly selected due to the abundance of
wildfires in hot seasons. Texas and Florida have been selected as they are
both subject to wildfires and storm events during different months.
Massachusetts and Illinois are selected first because those areas are
frequently subjected to bad weather, and second as they contain big cities
subject to traffic and local fire events alongside New York.

###  A.2 Keyword Filtering

This step selects the subset of filtered tweets related to a specific crisis
category of interest. Ideally, this task can be performed by a neural model
trained on a large set of tweets with labeled data indicating the categories.
However, as such a large amount of labeled data is unavailable, we rely on a
common approach of designing keywords. We carefully curate a keyword list for
each of our categories of interest by employing expert knowledge gathered
through over-viewing crisis stories in news, social media, and related big
disaster events of the past. Table 7 shows some example keywords used in each
of the crisis domains in CrisisLTLSum. Please note that these lists are
generic for each category and not specific to unique events. For instance,
instead of defining keywords specific to an event called “Hurricane Ida”, our
keywords list includes phrases such as “fallen tree”, “building collapse”, or
“storm”. The quality of the keywords list is crucial to the final quality of
the generated timelines, and we polish this list multiple times based on small
sets of experiments before using them for the final task. Each keyword can be
either a single word or multiple words. To avoid making a long list of
keywords and ensure that different lexical formats of the same word are still
considered in the keyword matching process, we maintain both a lower-cased and
a lemittized version of the keywords and the tweet’s text. If any of the
keywords exist in the text and in any of these formats, then the tweet is
considered related to the category. The multi-word keywords are not considered
n-grams but as an indication that all the words in the keyword should exist in
the text, even if not as a sequence. This approach will not be comprehensive
or exhaustive but rather representative of each crisis domain. Improving this
method to be more encompassing is an area for future research.

Wildfire | wildfire, bushfire, campfire, volcano erupt, forest fire, ash fall, vegetation fire
---|---
Local fire | house fire, structure fire, building fire, gas leak, fire alarm, tower burned
Storm | tornado, shelter, storm damage, building collapse, storm collapse, roof damage, fallen tree
Traffic | traffic delay, road blocked, car fire, crash, injury, rollover, accident, stalled vehicle
Table 7: Sample keywords for each crisis domain in CrisisLTLSum.

###  A.3 Clustering Hyper-Parameters

We use the following parameters shared among all different domains:
$s_{hashtag}$ is set to $0.2$ and $s_{dist}$ is $0.3$. The $sim_{threshold}$
and $time_{threshold}$ are set to $0.7$ and $15$ hours respectively. The
$time_{threshold}$ is reduced to $3$ hours to avoid merging various accidents
at different times but in the same location. The $min_{dist}$ and $max_{dist}$
are set to $0.4$ kilometer and $4$ kilometer for fire and traffic events while
having the larger range of $0.4$ and $10$ kilometers for the wildfire and
storm extraction. The $expiration_{threshold}$ is set to $15$ hours and
$tweet_{threshold}$ is $4$.

###  A.4 Automatic and Manual Cluster Merge

The online clustering approach may fail to properly relate some tweets due to
missing entities, clustering method hyper-parameters, or the difference in the
description of various angles of the same story. To address this, we merge
clusters by comparing all cluster heads with each other and combining those
with a higher similarity score than a threshold $s_{head}$. Additionally, we
use human feedback to merge clusters where their similarity score is below
$s_{head}$ but higher than a second hyper parameter $s_{head}^{min}$. This
process can compensate for a portion of the missing entities due to entity
extraction accuracy or tweets’ informal text.

###  A.5 Duplicate Removal

Here, the goal is to remove tweets from the same cluster with identical or
similar text. The duplicate removal relies on a fuzzy string sequence-matching
technique to compute the similarity between a pair of tweets. We simply go
over each tweet sorted in chronological order and remove the ones that match
any of the previous tweets in the same cluster with a matching score higher
than $d_{match}$.

###  A.6 Noise Removal

Although the keyword filtering step would reduce the number of tweets
unrelated to the category of interest, this step aims to further remove the
unrelated clusters generated from the pipeline. To do so, we use a neural
model for detecting the informativeness of tweets’ text by training it on the
available labeled data Alam et al. (2021a). As the definition of
informativeness in our domain differs from the data source, we further define
a mapping between their label set and the informativeness as we refer to it.
Hence, any fine-grained label related to personal emotions, prayers, or
donations is removed from the informative set. Since this process is very
category relevant and the existing labeled datasets do not cover all of our
categories of interest, we can only apply this step to those categories where
a representative subset exists for them in the available resources. We
formulate this task as a Boolean tweet classification task to predict the
tweet informativeness by applying a linear classification module on top of the
aggregation token of a transformer-based language model.

##  Appendix B Noisy Clusters Selection for Human Annotation

This section gives more details on how we select noisy clusters for human
annotation. Our goal is to select 1000 clusters that contain enough tweets
describing an evolving crisis event. In particular, we first identify a “seed
tweet” as the first observed post mentioning a local crisis event, then
roughly check whether the following tweets in the cluster contain updates
about the same event. Taking the example in Figure 1, we first see “a
vegetation fire happened” in the seed tweet, then see “plumes of smoke”,
“windy condition impacts fire control”. We select clusters across four crisis
domains, including wildfire, fire, storm, and traffic, depending on how
frequently each type of event happens in extracted noisy clusters.

| Train | Dev | Test
---|---|---|---
Domain | [1 – 5] | [6–12] | [13–25] | [26 –] | [1 – 5] | [6–12] | [13–25] | [26 –] | [1 – 5] | [6–12] | [13–25] | [26 –]
Wildfire | 125 | 101 | 42 | 35 | 14 | 11 | 5 | 4 | 36 | 28 | 12 | 10
Traffic | 113 | 66 | 15 | 9 | 13 | 8 | 2 | 2 | 32 | 21 | 3 | 3
Fire | 54 | 32 | 15 | 8 | 7 | 3 | 2 | 1 | 15 | 10 | 5 | 3
Storm | 14 | 30 | 18 | 12 | 2 | 4 | 3 | 2 | 4 | 2 | 0 | 0
Other | 11 | 5 | 0 | 1 | 2 | 1 | 0 | 0 | 5 | 9 | 6 | 4
Table 8: Aggregated data statistics based on crisis domains and timeline
lengths of the Train, Dev, and Test splits.

###  B.1 Sample noisy timelines

Figure 4 shows the noisy timeline of the same annotated example from Figure 1.
Figure 5, 6, and 7 show sample noisy timelines from other domains in
CrisisLTLSum.

Figure 4: Sample noisy timeline of a wildfire event. This is the same example
depicted in Figure 1. Figure 5: Sample noisy timeline of a traffic event.
Figure 6: Sample noisy timeline of a storm event. Figure 7: Sample noisy
timeline of a fire event.

##  Appendix C Detailed Experimental Setup

###  C.1 Timeline Extraction

##### BERT

We fine-tune BERT base uncased on a single GPU for 10 epochs with a learning
rate of 5e-5, batch size of 32, a seed of 42, and a maximum sequence length of
512. At the end of the fine-tuning, we pick the best checkpoint is based on
the performance on the Dev set.

##### BERT-GRU

For the BERT-GRU sequence labeling model, we use BERT base uncased to get the
contextual representation for each tweet. The GRU has one layer with a hidden
size of 128. The model was trained for 50 epochs with early stopping after
five epochs if the performance did not improve on the Dev set. We use a
learning rate of 5e-5, a batch size of 16, and a seed of 42.

###  C.2 Timeline Summarization

We fine-tune BART based on a single GPU for ten epochs with a learning rate of
5e-5, batch size of 16, a seed of 42, and a maximum target sequence length of
512. For DistillBART, we use the same hyperparameters. During inference, we
use beam search with a beam size of 4. At the end of the fine-tuning, we pick
the best checkpoint is based on the performance on the Dev set.

##  Appendix D Detailed Data Splits

Detail statistics on Train, Dev, and Test are shown in Table 8.

##  Appendix E Annotation Interface

Figure 8 \- 14 show the annotation interface for the Timeline Extraction &
Summarization task. Figure 15 \- 21 show the annotation interface for the
Summarization Quality Estimation (QE) task.

For the QE task, we first display the rating rubrics and examples to the
workers as shown in Figure 15 \- 17. To ensure the workers have a good
understanding of the QE dimensions listed in the rubrics, as shown in Figure
18, the workers are asked to pass a screening test before they can access the
quality rating part of the task interface. Once they answer the test question
with the correct answer, the rating interface will show up, as shown in Figure
19. In this page, we display the original tweet timeline on the left hand side
of the rating interface. Before displaying all the summaries and the rating
options, to make sure the workers go over the timeline carefully, we ask them
to use the length of the timeline to answer the last screening question before
conducting the rating task. The rating part of the interface is shown in
Figures 20 and 21.

Figure 8: The first page of our annotation interface for the Timeline
Extraction & Summarization task, which contains the task introduction and a
brief instruction. Figure 9: Step 1 - 2 of the annotation instructions in our
annotation interface for the Timeline Extraction & Summarization task Figure
10: Step 3 of the annotation instructions in our annotation interface for the
Timeline Extraction & Summarization task Figure 11: Step 4 and final hits of
the annotation instructions in our annotation interface for the Timeline
Extraction & Summarization task Figure 12: Step 1 - 2 of the task page in the
annotation interface for the Timeline Extraction & Summarization task Figure
13: Step 3 of the task page in the annotation interface for the Timeline
Extraction & Summarization task Figure 14: Completion page in the annotation
interface for the Timeline Extraction & Summarization task Figure 15:
Instruction page of the Summarization Quality Estimation Task part 1. Figure
16: Instruction page of the Summarization Quality Estimation Task part 2.
Figure 17: Instruction page of the Summarization Quality Estimation Task part
3. Figure 18: Instruction page of the Summarization Quality Estimation Task
part 4. Figure 19: Test page of the Summarization Quality Estimation Task.
Figure 20: Rating page of the Summarization Quality Estimation Task part 1.
Figure 21: Rating page of the Summarization Quality Estimation Task part 2.