# LLM Based Multi-Document Summarization
Exploiting Main-Event Biased Monotone Submodular Content Extraction

Litton J Kurisinkel, Nancy F. Chen
Institute for Infocomm Research, A*STAR, Singapore
litton_kurisinkel<EMAIL_ADDRESS>

###### Abstract

Multi-document summarization is a challenging task due to its inherent
subjective bias, highlighted by the low inter-annotator ROUGE-1 score of 0.4
among DUC-2004 reference summaries. In this work, we aim to enhance the
objectivity of news summarization by focusing on the main event of a group of
related news documents and presenting it coherently with sufficient context.
Our primary objective is to succinctly report the main event, ensuring that
the summary remains objective and informative. To achieve this, we employ an
extract-rewrite approach that incorporates a main-event biased monotone-
submodular function for content selection. This enables us to extract the most
crucial information related to the main event from the document cluster. To
ensure coherence, we utilize a fine-tuned Language Model (LLM) for rewriting
the extracted content into a coherent text. The evaluation using objective
metrics and human evaluators confirms the effectiveness of our approach, as it
surpasses potential baselines, demonstrating excellence in both content
coverage, coherence, and informativeness.

## Introduction

Presenting information in text format has been critical to the development of
human civilizations. Thus, text generation is an important field in artificial
intelligence and natural language processing, where the input to such natural
language generation models could take the form of text, graphs, images, or
database records (Zhang et al. 2020a; Song et al. 2018; Wiseman, Shieber, and
Rush 2018; Zhang et al. 2020b).

The pursuit of text-to-text generation in the NLP community has taken various
approaches and seen significant advancements, especially with the advent of
large language models (Grail, Perez, and Gaussier 2021). In the past, certain
methods have treated text-to-text generation as a search problem and utilized
discrete optimization techniques and greedy algorithms to find solutions
(Barzilay and Lapata 2006; Clarke and Lapata 2008). To improve the consistency
of greedy methods, some researchers have incorporated monotone submodular
functions (Lin and Bilmes 2011).

Operating in the discrete space has given these methods an advantage in terms
of exerting more control during text generation (Erten and Kazakov 2021).
However, they suffer from a critical drawback – the inability to effectively
learn from a large volume of training data and produce text that truly
resembles human-written text with good style.

Data-driven text generation techniques based on neural networks heavily rely
on training datasets to build and fine-tune their models (Xiao et al. 2021).
The reliability and efficacy of these models are inherently tied to the
quality and credibility of the datasets employed (Jain et al. 2020). In the
domain of text summarization, many researchers have utilized autocreated
datasets (Fabbri et al. 2019). Nevertheless, it is essential to acknowledge
the mounting concerns and potential criticisms regarding the reliability of
presently available datasets, particularly in the context of multi-document
summarization (Wolhandler et al. 2022).

Large language models are inherently designed to learn from large amounts of
training data using computations in a continuous space with dense
representations to produce text that cannot be discriminated from human-
written text (Touvron et al. 2023). However, memory-efficient LLMs suffer from
constraints on context length, which restricts their direct use for multi-
document summarization (Li 2023). Also, unlike traditional methods, neural
networks inherently provide fewer provisions to control intermediate
computations (Erdem et al. 2022). The factor of controllability is crucial
when it comes to real-world applications of text generation methods.
Additionally, lightweight large language models are yet to overcome the curse
of hallucinations (Azaria and Mitchell 2023). In such a situation, extract-
rewrite methods can be a relevant option to explore.

The inter-annotator ROUGE score of 0.4 between DUC-2004 reference summaries
may indicate the need for more objective methods in defining multi-document
summarization techniques 111https://duc.nist.gov/duc2004/. Our system aims to
summarize a cluster of documents related to the same main event. The objective
of our summarization is to report the main event with the most relevant
information, providing the necessary context to infer the main event
coherently. Our objective is conditioned by previous observations on event-
based discourse analysis for news articles (Van Dijk 2013).

In a nutshell, the contributions of the paper are:

  * •

A main event biased greedy method which formulates a more objective content
extraction method for Multi-Document Summarization (MDS) using a monotone
submodular function with linear components for coverage, diversity, and
coherence.

  * •

A fine-tuned Language Model (LLM) which takes the extracted content and re-
writes it to create a coherent summary.

The paper also introduces an annotated test set consisting of 30 clusters of
documents and corresponding main event-focused summaries.

Figure 1: Main Event biased Multi- Document Summarization: Context Extractor
is a Monotone Sub Modular Function

## Related Work

In the field of text summarization, there are two main approaches: extractive
and abstractive methods. Extractive methods involve directly selecting and
combining sentences from the original text, often yielding fluent outputs as
they use human-written content (Lin and Bilmes 2011; Wang et al. 2008). Some
extractive approaches have used parsed sentence structures to remove noise and
extract relevant content for the summary (Morita et al. 2013). To move towards
abstractive summarization, researchers introduced sentence compression
techniques, which aimed to reduce noise and condense information (Lin 2003;
Zajic et al. 2006; Almeida and Martins 2013). However, these methods faced
challenges in maintaining coherent reading for the summary reader. To address
the coherence issue, Christensen et al. (2013) proposed a method that
leverages corpus-level discourse graphs to achieve both structural and topical
coherence. During summary extraction, their system attempts to balance
salience (important content) and coherence. Despite their efforts in enhancing
coherence, these approaches have limitations when it comes to effectively
leveraging large datasets and the learning capabilities of neural networks.

Numerous recent studies have developed text-to-text generation methods based
on neural networks (Rush, Chopra, and Weston 2015; Chopra, Auli, and Rush
2016; Zhong et al. 2020; Liu 2019). Some of these works focus on generating
summaries from input documents (See, Liu, and Manning 2017; Wang et al. 2019;
Liu and Lapata 2019; Zhang et al. 2020a). The fundamental concept involves
training a neural network to automatically extract syntactic and semantic
features from the input text to generate the desired output. However, these
methods are less tractable and provide limited control over summary
attributes. Addressing these challenges will be crucial for advancing the
efficacy and versatility of text-to-text generation techniques.

Recent interest in NLP methods is sparked by the introduction of Large
Language models (LLMs) (Touvron et al. 2023; Penedo et al. 2023). These
language models possess a large set of parameters, equipping them with
exceptional learning capabilities. There are LLMs specifically trained for
various NLP tasks, including text-to-text generation and machine translation
(Ni et al. 2021). However, memory-efficient LLMs face limitations due to their
low context length (Chen et al. 2023) and may still experience hallucinations
(Azaria and Mitchell 2023). In this current work, we explore a more manageable
approach that can be applied in real-world scenarios. This approach involves
using a dedicated content extraction scheme to produce coherent output
summaries in a guided manner by rephrasing the extracted content.

## Problem Defenition

We define our problem in 3 steps.

$\displaystyle x_{\text{main}}$ $\displaystyle=E_{\text{main}}(X)$
$\displaystyle\text{context}({x_{\text{main}}})$
$\displaystyle=F(X,x_{\text{main}})$ $\displaystyle S_{\text{sum}}$
$\displaystyle=Z([x_{\text{main}},\text{context}({x_{\text{main}}})])$

Where $X$ represents the input set of documents that are related to a main
event which is created through event-based clustering. The function
$E_{\text{main}}$ is specifically designed to extract a sentence
$x_{\text{main}}$ from the collection $X$, and this sentence is intended to
represent the main event. Afterwards, the function $F$ takes on the task of
extracting a set of sentences referred to as
$\text{context}(x_{\text{main}})$. These extracted sentences provide the most
relevant contextual information for $x_{\text{main}}$, helping to give a
better understanding of the main event. To generate the final summary
$S_{\text{sum}}$, we employ the re-writing model $Z$. This model takes the
concatenation of the main event sentence $x_{\text{main}}$ with its
corresponding context $\text{context}(x_{\text{main}})$ to rewrite the
information contained in them to a final coherent summary.

## Method

Our method involves three main steps: main event extraction, context
extraction, and re-writing. These steps are depicted in Figure 1. We rely on
an event discourse-based approach to implement the main event extractor
$E_{\text{main}}$. The main event context extractor $F$ is a greedy method
based on an unsupervised monotone submodular function, while the re-writing
model $Z$ is a comparatively lightweight, large language model fine-tuned for
this purpose. We will explain each of these steps in detail in the subsequent
subsections.

###  $E_{\text{main}}$: Main Event Extraction

To extract the main event from articles, we rely on the discourse structure
proposed by Van Dijk (2013), which presents a main-event centric view of
discourse. According to this approach, all linguistic units in a document
share a discourse relation with the linguistic unit that represents the main
event. Choubey et al. (2020) studied the case where linguistic units are
sentences and introduced a method to identify discourse labels. We adopted the
same method to identify and extract main event sentences from a document. To
determine the main event of input set of documents X, we create a candidate
set of sentences by extracting main event sentence from each document in the
input set. From these candidates, we select the one with the maximum coverage
value as the main event of the document X as follows.

$\displaystyle x_{\text{main}}$
$\displaystyle=\operatorname*{argmax}_{x_{i}\in\text{candidates}}C(\\{x_{i}\\})$
(1)

$C(x_{i})$ is the coverage of the sentence $x_{i}$ which is computed using
Equation 5, and details of it will be explained in subsequent sections.

### F: A Greedy Method for Main event Context Extraction

The context extractor, denoted as F, focuses on extracting the most relevant
information needed to provide necessary context for the main event
$x_{\text{main}}$. F is an extractive summarization method that is biased
towards the main event. The problem of extractive summarization can be
represented as an argument maximization operation as follows.

$\displaystyle S_{\text{opt}}$
$\displaystyle=\underset{S}{\operatorname{argmax}}\,F(S)$ (2)

Where $S_{\text{opt}}$ is the optimum extractive summary and $F$ is a function
defined in the space of candidate summaries. Lin and Bilmes (2011) comes out
with the interesting observation that if $F$ is a monotone submodular
function, there exists a greedy method that can approximate the optimum
summary by a factor of 0.692.

$\begin{split}F(S_{\text{GREEDY}})&\geq\left(1-\frac{1}{e}\right)F(S_{\text{opt}})\\\
&\approx 0.632F(S_{\text{opt}})\end{split}$ (3)

Where $S_{\text{GREEDY}}$ is a summary computed using corresponding greedy
method. Inspired by this relation, we formulate a $F$ as follows.

$\displaystyle F(S)$ $\displaystyle=C(S)+\lambda_{1}\cdot
D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$ (4)

Where $C$, $D$, and $B_{\text{main}}$ are functions that measure the topical
coverage, topical diversity, and main event-based importance, respectively, of
candidate summary $S$. Ensuring that these linear components are monotone
submodular will consequently ensure that $F$ is submodular. The rest of this
section explains each one of them in detail.

####  $C(S)$ : Content Coverage

$C(S)$ is the set function that estimates the extent to which a candidate
summary $S$ is a topical representative of the input set of documents $X$. We
use an enhanced version of the topical coverage function proposed by Lin and
Bilmes (2011), which is based on dense sentence representations, to estimate
coverage.

$\displaystyle C(S)$ $\displaystyle=\sum_{i\in
U}\min\\{c_{i}(S),\alpha*c_{i}(U)\\}$ (5)

where $\alpha$ is an empirically optimized constant $U$ represents the set of
all sentences in the input set of documents and,

$\displaystyle c_{i}(S)=\sum_{j\in S}Sim(i,j)$ (6)

Where $Sim$ computes the cosine similarity between S-BERT representations of
argument sentences (Reimers and Gurevych 2019).

####  $D(S)$:Content Diversity

Our diversity function, designed to measure the content diversity of candidate
summaries, is a variant of the diversity function proposed by Lin and Bilmes
(2011). The process involves clustering the input set of sentences using the
affinity propagation algorithm (Dueck 2009) with the similarity metric $Sim$.
We then utilize the information regarding these clusters to compute the
content diversity of summaries as follows.

$\displaystyle D(S)=\sum_{i=1}^{K}\sqrt{\sum_{j\in P_{i}\cap
S}\frac{c_{i}(U)}{N}}$ (7)

Where $N$ be the total number of sentences in $U$, and let $P_{i}$ be the set
of sentences in cluster $i$. Thus, the diversity function encourages summaries
to have representations for more number of clusters.

Dataset | Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---
Multi- News | PEGASUS | 36.5 | 10.5 | 18.7
BART | 27.3 | 6.2 | 15.1
LED | 17.3 | 3.7 | 10.4
PRIMERA | 42.0 | 13.6 | 20.8
$C(S)+\lambda_{1}\cdot D(S)$ | 40.5 | 12.7 | 19.3
$C(S)+\lambda_{1}\cdot D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$ | 43.0 | 14.2 | 21.3
WCEP | PEGASUS | 33.2 | 12.7 | 23.8
BART | 20.2 | 5.7 | 15.3
LED | 18.8 | 5.4 | 14.7
PRIMERA | 28.0 | 10.3 | 20.9
$C(S)+\lambda_{1}\cdot D(S)$ | 32.5 | 11.3 | 20.7
$C(S)+\lambda_{1}\cdot D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$ | 33.7 | 12.9 | 23.7
Main Event | PEGASUS | 32.1 | 9.5 | 16.1
BART | 25.3 | 6.1 | 14.0
LED | 16.0 | 4.1 | 9.2
PRIMERA | 42.3 | 12.8 | 19.1
$C(S)+\lambda_{1}\cdot D(S)$ | 41.3 | 13.6 | 20.5
$C(S)+\lambda_{1}\cdot D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$ | 49.9 | 22.7 | 27.9
Table 1: Text Summarization Performance Comparison using ROUGE Metrics: Our
Method with Main-Event Bias Outperforms Peer Systems

####  $B_{\text{main}}(S)$:Main Event Based Relevance

Our approach mainly focuses on reporting the main event with essential details
in the summary. Therefore, $B_{\text{main}}$ is a crucial component that
computes the relevance of a candidate summary with respect to the main event.
One way to compute this could be based on the discourse label of sentences in
the input document with respect to the main event, as per the news discourse
structure suggested by (Choubey et al. 2020). This approach ensures that the
summary emphasizes the main event and includes the necessary information
related to it.

$\displaystyle B_{\text{main}}$ $\displaystyle=\sum_{s\in
S}\text{P}\left(s/\text{discourse}(s,x_{\text{main}})\right)$ (8)

Where P computes the probability of sentence $s$ being in the summary with
respect to its discourse label $\text{discourse}(s,x_{\text{main}})$. However,
it is extremely difficult to identify these discourse labels in the context of
multi-document, as it requires extensive effort. For this reason, we
investigated an approach that could implicitly compute the relevance of a
sentence with respect to the main event without explicitly identifying
discourse labels. Consequently, we found a method based on co-occurrence
probability, in which main event-based relevance is computed as follows.

$\displaystyle B_{\text{main}}(S)=\sum_{i\in S}Coc(i,x_{\text{main}})$ (9)

Where $Coc$ is a neural network that measures the co-occurrence probability of
argument sentences. Given sentence vectors $s_{1}$ and $s_{2}$ for argument
sentences, the network computes $[s_{1},s_{2}]$, $s_{1}-s_{2}$, $s_{1}*s_{2}$,
and $|s_{1}-s_{2}|$ (Xu et al. 2019). These features are concatenated and fed
into an MLP layer. The model is made order-agnostic by training a forward
model with input $(s_{1},s_{2})$ and a backward model with input
$(s_{2},s_{1})$, using the same architecture but different parameter sets. The
co-occurrence score is computed as the average of the two models. The loss for
a triplet $\mathcal{T}=(s_{i},s_{p},s_{n})$ with positive pair $(s_{1},s_{p})$
and negative $(s_{1},s_{n})$ is computed as follows,

$\displaystyle\mathcal{L}_{\mathcal{T}}=\max(0,m+Coc(\mathbf{s_{1}},\mathbf{s_{p}})-Coc(\mathbf{s_{1}},\mathbf{s_{n}}))$
(10)

Where $m$ is the margin.

### Summarization

Lin and Bilmes (2011) deductively showed that $C(S)$ and $D(S)$ are
monotonically submodular. $B_{\text{main}}(S)$ is monotonically submodular as
$Coc(i,x_{\text{main}})$ is constant for all sentences in the input document
set. Consequently, $F$ is monotonically submodular. We start our corresponding
greedy method with a maximum weight selection scheme to extract
$S_{\text{GREEDY}}$, beginning with the singleton set ${x_{\text{main}}}$.
This approach allows the extracted summary $S$ to effectively approximate
[$x_{\text{main}},\text{context}({x_{\text{main}}})$]. The number of sentences
to be extracted, $N$, is decided using the variance $\sigma^{2}$ among
similarity values between input sentences measured using $sim$.

$\displaystyle N=$ $\displaystyle\lfloor k+c\cdot\sigma^{2}\rfloor$

Where $k$ and $c$ are constants which are optimized emprically.

### Z: Re- writing Model

The extracted set of summary sentences needs to be rewritten into a coherent
text using a large language model fine-tuned for this purpose. We created a
parallel dataset consisting of the extracted set of summary sentences and
their corresponding coherently written versions using the method from the
multi-news dataset (Fabbri et al. 2019). To create the dataset, we used the
alignment method employed by Wolhandler et al. (2022). We fine-tuned the
flan-t5-xl model222https://huggingface.co/google/flan-t5-xl, which is
specifically pre-trained for text-to-text generation, and introduced the ’re-
write’ prompt during training. The T5 model is lightweight and requires less
compute for training and inference.

## Experiments

We conducted experiments to evaluate various attributes of extracted
summaries, such as content coverage, and coherence. Additionally, we evaluated
our submodels, which estimate the sentence co-occurrence score of sentences
and rewrite the extracted summary. This section provides further details about
our experimental settings and the datasets used for evaluations.

### Co-Occurance Model

We collected 45,000 news documents and extracted the first three sentences
from each document to constitute our pre-training dataset. Subsequently, we
trained the model using multi-news summaries (Fabbri et al. 2019). Any pair of
documents taken from the same piece of text is a positive pair, and any pair
of sentences taken from two different texts represents a negative pair. For
each positive pair, we choose a corresponding negative pair to form the
triplet explained in Equation 10. The cardinality of our training,
development, and test sets is 45,000, 1800, and 1800, respectively. We set the
margin $m$ to 5.4 in Equation 10. The results are shown in Table 2. 0.77
serves as a dependable F-Measure value suitable for use as a sub-component.

Precision | Recall | F-Measure
---|---|---
0.93 | 0.70 | 0.77
Table 2: Sentence Co- Occurance Results

### Fine-tuning LLM for re- writing

We use the flan-t5-xl model333https://huggingface.co/google/flan-t5-xl for the
purpose of re-writing. As explained earlier in the paper, we created a dataset
by aligning summary sentences in the Multi-news dataset to source sentences.
We created around 3000 parallel records for fine-tuning, 450 for development,
and 450 for testing. The fine-tuned re-writing model yielded a BLEU score of
0.30.

### Text Summarization

We conducted experiments to evaluate our summaries on the dimensions of
content coverage and coherence. We relied on objective metrics such as
ROUGE-1, ROUGE-2 and ROUGE-L for estimating content coverage while on human
evaluation for estimating coherence.

#### Data:

We evaluated our approach for summarization using public datasets such as
Multi-news (Fabbri et al. 2019) and WCEP test sets (Ghalandari et al. 2020).
Multi-news contains a cluster of related news documents as input and
corresponding summaries. The WCEP dataset for multi-document summarization
(MDS) consists of short, human-written summaries about news events, obtained
from the Wikipedia Current Events Portal (WCEP), each paired with a cluster of
news articles associated with an event. In addition to these datasets, we
created a dataset in which summaries are written to report the main event. The
particular dataset consists of 30 clusters of documents. Each cluster of
documents is related to a single main event and contains two to four
documents. Our annotators are two post-graduate research assistants who have
sufficiently long experience in data annotation for NLP. They were asked to
annotate the event labels of each sentence in each document as per the
annotation scheme. Subsequently, they were asked to report the main event with
more relevant context. While writing context information, they were asked to
consider the discourse labels of individual sentences as guiding information.

#### Settings:

The constants $\alpha$, $\lambda_{1}$, $\lambda_{2}$, $k$, and $c$ are
optimized using grid search to attain the maximum ROUGE-L + ROUGE-2 score on
the development set for both the WCEP and Multi-News datasets collectively.

#### Results

Our results are shown in the Table 1. The table show the comparison of our
approach with zero/ a few shot results of pre- trained models namely : BART
(Lewis et al. 2019), PEGASUS (Zhang et al. 2020a), Longformer-Encoder
Decoder(LED) (Beltagy, Peters, and Cohan 2020) and PRIMERA (Xiao et al. 2021).
The comparison is restricted to zero/ a few shot results for a level
comparison as our approach is unsupervised as far as summarization is
concerned. For experiments we included our approach with main-event bias
($C(S)+\lambda_{1}\cdot D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$) and without
main-event bias ($C(S)+\lambda_{1}\cdot D(S)$). Our approach consistently
performed better than base models in all the datasets in terms of all the
three metrics considered. Base models lack an explicit control of summary
content selection and are more probable to hallucinate. On the other hand, our
method employs an explicit content extraction mechanism so that re- writing is
more guided. This could result in more reliable content and less prone to
hallucination. The result increased result in our dataset is obvious as our
method is specifically designed the main event with necessary context.

Figure 2: Human Evaluation for Coherence
---
System Summary
Two women have been charged with criminal damage after climate change
protesters threw soup over Vincent van Gogh’s painting Sunflowers at London’s
National Gallery, British police said on Saturday (Oct 15). Van Gogh’s
Sunflowers, valued at roughly US $80 million, had no connection to climate
change, according to authorities. A video posted by the Just Stop Oil campaign
group, which has been conducting protests for the past two weeks in the
British capital, showed two of its activists on Friday throwing tins of Heinz
tomato soup onto the painting.
Reference Summary
Two women have been charged with criminal damage after climate change
protesters threw soup over Vincent van Gogh2̆019s painting Sunflowers at
London’s National Gallery, British police said on Saturday (Oct 15). A video
posted by the Just Stop Oil campaign group, which has been holding protests
for the last two weeks in London,0̆0a0showed two of its activists on Friday
throwing tins of Heinz tomato soup over the painting, one of the treasures of
the museum’s collection. Police said the two women, aged 21 and 20, would
appear later at Westminster Magistrates’ Court charged with c̈riminal damage
to the frame of van Gogh’s Sunflowers painting
Table 3: Summaries generated by our system and corresponding reference
summaries

### Human Evaluation for Coherence

To evaluate coherence, we selected four human evaluators who are postgraduate
students in linguistics. We randomly assembled a sample set of output
summaries, comprising 45 summary sets. Each summary set contains summaries
generated by the following settings: $BART$, $PRIMERA$, $C(S)+\lambda_{1}\cdot
D(S)$, and $C(S)+\lambda_{1}\cdot D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$.
The summaries were presented to the evaluators in a randomized order to
prevent any potential bias. The evaluators were then asked to identify the
most coherent summary among those listed. They were instructed to assess
coherence based on discourse connections using linguistic cues and to consider
topical continuity between neighboring sentences in the summaries. The results
are shown in the Figure 2. The settings utilizing large language models (LLMs)
were overwhelmingly preferred by the evaluators. This preference is
unsurprising, given that LLMs are more reliable for text generation compared
to text generation networks with a relatively smaller number of parameters.
Even after incorporating LLMs into the text generation process,
$C(S)+\lambda_{1}\cdot D(S)$, the setting without main event bias was selected
a lesser number of times by the evaluators in comparison with
$C(S)+\lambda_{1}\cdot D(S)+\lambda_{2}\cdot B_{\text{main}}(S)$. Bias term
$B_{\text{main}}(S)$ encourages sentences in the extractive summaries to be
more semantically aligned with the initially added main event sentence. When
the semantic relatedness between extracted sentences is high, a coherent
summary can be achieved when re-written by an LLM as semantic relatedness
contributes to coherence (Aletras and Stevenson 2013). Table 3 contains a
sample system-generated summary with high ROUGE scores, along with its
corresponding reference summary. The system-generated summary demonstrates a
substantial level of content overlap with the reference summary and maintains
topical continuity that is quite similar to the reference summary.

## Conclusion

In this current work, we introduced a main event-focused method for multi-
document news summarization. Our approach consistently outperformed peer
systems in terms of ROUGE metrics. Summaries generated by the method exhibited
better coherence, as per human evaluation. Future work could explore improved
counterparts for each component of the method, namely main event extraction,
context extraction, and rewriting.

## References

  * Aletras and Stevenson (2013) Aletras, N.; and Stevenson, M. 2013.  Evaluating topic coherence using distributional semantics.  In _Proceedings of the 10th international conference on computational semantics (IWCS 2013)–Long Papers_ , 13–22.
  * Almeida and Martins (2013) Almeida, M. B.; and Martins, A. F. 2013.  Fast and Robust Compressive Summarization with Dual Decomposition and Multi-Task Learning.  In _ACL (1)_ , 196–206.
  * Azaria and Mitchell (2023) Azaria, A.; and Mitchell, T. 2023.  The internal state of an llm knows when its lying.  _arXiv preprint arXiv:2304.13734_.
  * Barzilay and Lapata (2006) Barzilay, R.; and Lapata, M. 2006.  Aggregation via set partitioning for natural language generation.  In _Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics_ , 359–366. Citeseer.
  * Beltagy, Peters, and Cohan (2020) Beltagy, I.; Peters, M. E.; and Cohan, A. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Chen et al. (2023) Chen, S.; Wong, S.; Chen, L.; and Tian, Y. 2023.  Extending context window of large language models via positional interpolation.  _arXiv preprint arXiv:2306.15595_.
  * Chopra, Auli, and Rush (2016) Chopra, S.; Auli, M.; and Rush, A. M. 2016.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 93–98.
  * Choubey et al. (2020) Choubey, P. K.; Lee, A.; Huang, R.; and Wang, L. 2020.  Discourse as a function of event: Profiling discourse structure in news articles around the main event.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Christensen et al. (2013) Christensen, J.; Soderland, S.; Etzioni, O.; et al. 2013.  Towards coherent multi-document summarization.  In _Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies_ , 1163–1173.
  * Clarke and Lapata (2008) Clarke, J.; and Lapata, M. 2008.  Global inference for sentence compression: An integer linear programming approach.  _Journal of Artificial Intelligence Research_ , 31: 399–429.
  * Dueck (2009) Dueck, D. 2009.  _Affinity propagation: clustering data by passing messages_.  University of Toronto Toronto, ON, Canada.
  * Erdem et al. (2022) Erdem, E.; Kuyu, M.; Yagcioglu, S.; Frank, A.; Parcalabescu, L.; Plank, B.; Babii, A.; Turuta, O.; Erdem, A.; Calixto, I.; et al. 2022.  Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning.  _Journal of Artificial Intelligence Research_ , 73: 1131–1207.
  * Erten and Kazakov (2021) Erten, C.; and Kazakov, D. L. 2021.  Ontology Graph Embeddings and ILP for Financial Forecasting.  In _Inductive Logic Programming, Proceedings of the 30th International Conference:_. Springer.
  * Fabbri et al. (2019) Fabbri, A. R.; Li, I.; She, T.; Li, S.; and Radev, D. R. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  _arXiv preprint arXiv:1906.01749_.
  * Ghalandari et al. (2020) Ghalandari, D. G.; Hokamp, C.; Pham, N. T.; Glover, J.; and Ifrim, G. 2020.  A large-scale multi-document summarization dataset from the Wikipedia current events portal.  _arXiv preprint arXiv:2005.10070_.
  * Grail, Perez, and Gaussier (2021) Grail, Q.; Perez, J.; and Gaussier, E. 2021.  Globalizing BERT-based transformer architectures for long document summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 1792–1810.
  * Jain et al. (2020) Jain, A.; Patel, H.; Nagalapatti, L.; Gupta, N.; Mehta, S.; Guttula, S.; Mujumdar, S.; Afzal, S.; Sharma Mittal, R.; and Munigala, V. 2020.  Overview and importance of data quality for machine learning tasks.  In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_, 3561–3562.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li (2023) Li, Y. 2023.  Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering.  _arXiv preprint arXiv:2304.12102_.
  * Lin (2003) Lin, C.-Y. 2003.  Improving summarization performance by sentence compression: a pilot study.  In _Proceedings of the sixth international workshop on Information retrieval with Asian languages-Volume 11_ , 1–8. Association for Computational Linguistics.
  * Lin and Bilmes (2011) Lin, H.; and Bilmes, J. 2011.  A class of submodular functions for document summarization.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , 510–520.
  * Liu (2019) Liu, Y. 2019.  Fine-tune BERT for extractive summarization.  _arXiv preprint arXiv:1903.10318_.
  * Liu and Lapata (2019) Liu, Y.; and Lapata, M. 2019.  Text summarization with pretrained encoders.  _arXiv preprint arXiv:1908.08345_.
  * Morita et al. (2013) Morita, H.; Sasano, R.; Takamura, H.; and Okumura, M. 2013.  Subtree Extractive Summarization via Submodular Maximization.  In _ACL (1)_ , 1023–1032. Citeseer.
  * Ni et al. (2021) Ni, J.; Ábrego, G. H.; Constant, N.; Ma, J.; Hall, K. B.; Cer, D.; and Yang, Y. 2021.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  _arXiv preprint arXiv:2108.08877_.
  * Penedo et al. (2023) Penedo, G.; Malartic, Q.; Hesslow, D.; Cojocaru, R.; Cappelli, A.; Alobeidli, H.; Pannier, B.; Almazrouei, E.; and Launay, J. 2023.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Rush, Chopra, and Weston (2015) Rush, A. M.; Chopra, S.; and Weston, J. 2015.  A neural attention model for abstractive sentence summarization.  _arXiv preprint arXiv:1509.00685_.
  * See, Liu, and Manning (2017) See, A.; Liu, P. J.; and Manning, C. D. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * Song et al. (2018) Song, L.; Zhang, Y.; Wang, Z.; and Gildea, D. 2018.  A graph-to-sequence model for AMR-to-text generation.  _arXiv preprint arXiv:1805.02473_.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Van Dijk (2013) Van Dijk, T. A. 2013.  _News as discourse_.  Routledge.
  * Wang et al. (2008) Wang, D.; Li, T.; Zhu, S.; and Ding, C. 2008.  Multi-document summarization via sentence-level semantic analysis and symmetric matrix factorization.  In _Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval_ , 307–314. ACM.
  * Wang et al. (2019) Wang, Q.; Li, B.; Xiao, T.; Zhu, J.; Li, C.; Wong, D. F.; and Chao, L. S. 2019.  Learning deep transformer models for machine translation.  _arXiv preprint arXiv:1906.01787_.
  * Wiseman, Shieber, and Rush (2018) Wiseman, S.; Shieber, S. M.; and Rush, A. M. 2018.  Learning Neural Templates for Text Generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 3174–3187.
  * Wolhandler et al. (2022) Wolhandler, R.; Cattan, A.; Ernst, O.; and Dagan, I. 2022.  How” Multi” is Multi-Document Summarization?  _arXiv preprint arXiv:2210.12688_.
  * Xiao et al. (2021) Xiao, W.; Beltagy, I.; Carenini, G.; and Cohan, A. 2021.  PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization.  _arXiv preprint arXiv:2110.08499_.
  * Xu et al. (2019) Xu, P.; Saghir, H.; Kang, J. S.; Long, T.; Bose, A. J.; Cao, Y.; and Cheung, J. C. K. 2019.  A cross-domain transferable neural coherence model.  _arXiv preprint arXiv:1905.11912_.
  * Zajic et al. (2006) Zajic, D. M.; Dorr, B.; Lin, J.; and Schwartz, R. 2006.  Sentence compression as a component of a multi-document summarization system.  In _Proceedings of the 2006 Document Understanding Workshop, New York_.
  * Zhang et al. (2020a) Zhang, J.; Zhao, Y.; Saleh, M.; and Liu, P. 2020a.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , 11328–11339. PMLR.
  * Zhang et al. (2020b) Zhang, Y.; Wang, X.; Xu, Z.; Yu, Q.; Yuille, A.; and Xu, D. 2020b.  When radiology report generation meets knowledge graph.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 12910–12917.
  * Zhong et al. (2020) Zhong, M.; Liu, P.; Chen, Y.; Wang, D.; Qiu, X.; and Huang, X.-J. 2020.  Extractive Summarization as Text Matching.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 6197–6208.