# CADS: A Systematic Literature Review on the Challenges of Abstractive
Dialogue Summarization

Frederic Kirstein<EMAIL_ADDRESS>
Jan Philip Wahle<EMAIL_ADDRESS>
Bela Gipp<EMAIL_ADDRESS>
Terry Ruas<EMAIL_ADDRESS>
Georg-August University Göttingen, Papendiek 14,
Göttingen, NRW, 37073, Germany

###### Abstract

Abstractive dialogue summarization is the task of distilling conversations
into informative and concise summaries. Although focused reviews have been
conducted on this topic, there is a lack of comprehensive work that details
the core challenges of dialogue summarization, unifies the differing
understanding of the task, and aligns proposed techniques, datasets, and
evaluation metrics with the challenges. This article summarizes the research
on Transformer-based abstractive summarization for English dialogues by
systematically reviewing 1262 unique research papers published between 2019
and 2024, relying on the Semantic Scholar and DBLP databases. We cover the
main challenges present in dialog summarization (i.e., language, structure,
comprehension, speaker, salience, and factuality) and link them to
corresponding techniques such as graph-based approaches, additional training
tasks, and planning strategies, which typically overly rely on BART-based
encoder-decoder models. We find that while some challenges, like language,
have seen considerable progress, mainly due to training methods, others, such
as comprehension, factuality, and salience, remain difficult and hold
significant research opportunities. We further investigate how these
approaches are typically assessed, covering the datasets for the subdomains of
dialogue (e.g., meeting, customer service, and medical), the established
automatic metrics (e.g., ROUGE), and common human evaluation approaches for
assigning scores and evaluating annotator agreement. We observe that only a
few datasets with community interest span across all subdomains. Despite its
limitations, the ROUGE metric is the most commonly used, while human
evaluation, considered the gold standard, is frequently reported without
sufficient detail on inner-annotator agreement and annotation guidelines.
Additionally, we discuss the possible implications of the recently explored
large language models and conclude that despite a potential shift in relevance
and difficulty, our described challenge taxonomy remains relevant.

##  1 Introduction

Figure 1: Overview of the big six challenges in dialogue summarization, a
short description of the challenges, and the estimation of their progress
derived from related sub-challenges. Green means mostly mitigated, orange
means good progress, and red stands for marked challenges still exist.

Abstractive dialogue summarization, a task within Natural Language Processing
(NLP) and text summarization, entails condensing key information from
conversations into succinct and coherent summaries (?). This sub-field of text
summarization is gaining prominence and is relevant for various real-world
scenarios, including customer service (e.g., social media (?) and e-commerce
(?)), healthcare (?), daily life (?), meetings (?), and open-domain
conversations (e.g., online-chat (?)). The relevance of this task, which
offers insights into discussed concerns or issues, is thereby driven by the
increase in digital conversations in these scenarios (?). While the common,
manual approach to dialogue summarization is considered to be prone to human
errors and consumes significant time and effort, automatic dialogue
summarization may effectively reduce this overhead on the user side (?).

Diverging from traditional text summarization, which typically focuses on
formal and linear content such as news articles (?) or scientific publications
(?), dialogue summarization confronts unique challenges (?): conversational
text is inherently dynamic, interactive, and non-linear (?), marked by
verbosity, repetition, and informality. Salient information is often
distributed across multiple speakers, mixed with casual and off-topic remarks
(?). Additionally, the use of elliptical and fragmented sentences,
characterized by, e.g., incomplete thoughts and context-specific
abbreviations, further hinders the summarization process. These complexities
make traditional text summarization approaches less effective, which are not
trained to bridge the gap between edited text and dialogue and struggle to
transfer learned patterns through few-shot learning (?).

Despite advancements achieved by leveraging and adapting general-purpose pre-
trained neural language models (?, ?), and the recent trend to explore Large
Language Models (LLMs) for this task (?), there remains a notable gap in the
field’s foundational understanding of the challenges when processing
conversation transcripts. While research papers introduce techniques
addressing similar challenges, their definitions and interpretations of these
challenges can vary markedly. For instance, while some describe the informal
and ungrammatical nature of spoken language as a language challenge (?, ?),
others frame the challenge as different individual styles and unstructured
expressions (?, ?). Both points of view are thereby valid but, on their own,
focus just on a subpart of the language challenge and blur the understanding
of it. Current surveys leave out the challenges in dialogue summarization and
their variations in definitions, focusing instead on details on datasets,
metrics, and techniques. As a consequence, there is no clear understanding of
which aspects of dialogue summarization are well-addressed and where
significant research potential lies. Our systematic review consolidates the
existing knowledge researched in the field considering Transformer-based
models, aiming to provide a comprehensive overview of the challenges and their
progress. We further consider the center interest in LLMs and discuss their
impact on the relevance of the dialogue summarization challenges.

This article focuses on four key areas, namely challenges, dataset overviews,
techniques, and evaluation methods. In challenges, we categorize known
challenges for Transformer-based models found when dealing with dialogue
transcripts into six broader challenge blocks, i.e., language, structure,
comprehension, speaker, salience, and factuality. Our proposed taxonomy is
displayed in Figure 1, including the six challenge blocks, a short
description, and exemplary sub-challenges of the challenge blocks. In datasets
overview, our review displays used datasets focused on dialogue summarization,
organized by their subdomain, and further an overview of techniques to
generate datasets to cope with the existing data scarcity artificially and how
to optimize data usage. In techniques of dialogue summarization, we then match
97 techniques proposed since 2019 to the six challenge blocks. Furthermore, in
evaluation methods, we give an overview of typical evaluation metrics used,
spanning count-based metrics (e.g., ROUGE (?)), model-based (e.g., BERTScore
(?)), QA-based (e.g., QuestEval (?)), and human evaluation metrics for
performance and annotator agreement. This systematic review contributes to a
better understanding of the dialogue summarization problem, unifies the
inherent definitions, overviews established techniques, and demonstrates the
scarcity of datasets and fitting evaluation metrics.

Organization of the Literature Review. The remainder of this review is
structured as follows. We introduce the current state of dialogue
summarization in Section 1.1 and show our methodology in Section 2. In Section
3, we provide the general problem definition and our challenges taxonomy
consisting of six challenge blocks inherent in conversation scripts, further
linking proposed approaches to handle the challenges. Section 4 overviews
prominent datasets while the display of evaluation approaches in Section 5
helps readers choose suitable indices to evaluate the effectiveness of a
model. Finally, Section 6 discusses future research directions followed by
conclusions in Section 7. All resources for our review are publicly
available.111https://anonymous.4open.science/r/LitRev-DialogueSum/

###  1.1 Existing Reviews in Dialogue Summarization

Comprehensive literature reviews on dialogue summarization remain scarce, with
notable contributions by ? (?) and ? (?). Existing surveys mainly explore the
subdomains, their datasets, techniques, and metrics within dialogue
summarization. They provide a high-level overview and categorization of
prominent subdomains like meeting, email, chat, medical, and customer service
but often neglect niche areas such as e-commerce and debates (?). On the
datasets, ? (?) provide a framework linking linguistic dialogue types (?)
(e.g., persuasion, negotiation, inquiry) with established datasets, offering
insights into dataset suitability. ? (?) analyze dialogue summarization
techniques up to early 2022, categorizing techniques by conversational context
and component modeling (e.g., speaker, addressee, and utterance modeling to
solve the ’Who says what to whom’ paradigm), covering a subset of the
conversation challenges indirectly. ? (?) extend this analysis with techniques
from 2023, exploring approaches like hierarchical models for long inputs (?),
feature injection for enhanced understanding, auxiliary tasks for broader
training objectives, and data augmentation from related tasks. Regarding
evaluations, works on dialogue summarization leverage automatic metrics from
traditional document summarization, with the study by ? (?) discussing their
individual strengths and limitations.

While existing surveys provide an understanding of dialogue summarization
regarding subdomains, datasets, techniques, and metrics, the field misses a
link between these categories through a common framework, such as the
challenges inherent in processing dialogues. Our review addresses this gap by
introducing a comprehensive taxonomy of dialogue challenges and categorizing
recent techniques until 2024 based on these challenges (Section 3). This
approach highlights well-explored areas and points out gaps in current
research. We analyze datasets (Section 4) and evaluation metrics (Section 5),
linking them to the identified challenges. Additionally, we discuss the
implications of the shift from smaller encoder-decoder models such as BART (?)
to LLMs and the implications for the relevance of the challenges with this new
backbone model type (Section 6). Our review concentrates exclusively on
dialogue summarization and does not encompass broader areas such as general
text summarization (?) or dialogue generation (?).

##  2 Methodology

We use the PRISMA checklist (?) for organizing our literature review, as it is
an established approach to set up a comprehensive systematic literature review
while reducing the potential for incomplete data and biases in content
selection and presentation (?). This ensures that our approach is structured,
transparent, and reproducible.

Figure 2: An overview of the different stages of our methodology and their
inherent sub-stages.

Our methodology comprises two stages as detailed in Figure 2.

Retrieval Stage. We retrieve literature automatically and keyword-based from
two established academic databases, i.e., Semantic Scholar and DBLP, using
specific retrieval criteria and defining keyword queries for systematic
searching. The retrieved works are then automatically de-duplicated and saved
to a list.

Manual Stage. The second stage involves manually screening full documents
using strict inclusion and exclusion criteria. In this stage, we follow
additional best practices for manual filtering beyond those in the PRISMA
checklist. These include tracking specific items such as problem category,
challenges discussed, and proposed technique category (?, ?) and guidelines
for human annotators (?, ?).

We evaluate 1262 unique papers from 2019 until 2024, offering an up-to-date
perspective on the state of the art in dialogue summarization using
Transformer-based architectures. We decide against including references from
selected papers that did not appear in our crawl as additional sources to
maintain an unbiased selection process and avoid complicating the stopping
criterion for literature inclusion. Considering our exclusion criteria, such
as multilingualism, non-abstractive, and non-Transformer-based approaches, we
select 133 papers for our literature review.

###  2.1 Stage 1: Retrieving Candidate Documents

Information Sources. Our primary data sources are Semantic
Scholar222https://www.semanticscholar.org/ and DBLP333https://dblp.org/
(DataBase systems and Logic Programming), both recognized for extensive
coverage in computer science (?, ?). While Semantic Scholar allows for
advanced search functionalities and complex queries (?, ?, ?), DBLP is the
most comprehensive database for computer science publications, frequently used
in other surveys as sole source (?, ?, ?) and encompassing major peer-reviewed
journals and conference proceedings. This dual-source strategy ensures
thorough retrieval and coverage specific to dialogue summarization,
strengthening the certainty and reliability of our evidence base. To ensure
our literature review remains up-to-date, we regularly re-crawl databases
while conducting the review to incorporate the latest publications into our
analysis. 444Last crawled on March 25, 2024. While we acknowledge the
potential for these databases to miss the most recent uploads, our strategy
minimizes this risk. We exclude databases such as Google Scholar, which overly
returned grey literature, and Web of Science, which emphasized extractive
summarization techniques during our pre-testing, to maintain a focus on peer-
reviewed literature in abstractive dialogue summarization. We design an
automatic pipeline to retrieve query-related works and remove duplicates. We
limit the number of results per query to the top 100 based on the relevance
ranking provided by the platforms, which was rarely required. We recognize
that the ranking systems may be biased towards prestigious journals or highly
cited authors. Hence, we retrieve all papers published after June 2023, not
relying on the ranking systems to account for their limited time to gather
citations.

Data Collection Process. Our search strategy uses key terms related to
dialogue summarization identified during our initial related work analysis
(Section 1.1). Anchor terms such as ’dialogue summarization’ and ’multi-party
conversation’ are combined with specific search terms such as ’technique’. We
use four anchor terms and 28 search terms, resulting in 112 search queries for
each database. We present the complete list and search keywords in our
repository555https://anonymous.4open.science/r/LitRev-DialogueSum/ . Employing
a Python-based pipeline (Section 8), we systematically extract documents from
Semantic Scholar and DBLP APIs, merging and unifying the results into tabular
data. Between January 2019 and March 2024, we retrieved 732 publications from
DBLP and 968 from Semantic Scholar. After removing 438 duplicates identified
across both sources, we have 1262 unique publications at the end of the stage.
These results are tagged with their respective queries and compiled into a CSV
file for subsequent selection and analysis.

###  2.2 Stage 2: Manual Filtering

Eligibility Criteria. For our manual filtering process, we define eligibility
criteria to ensure the focus of our literature review. We look at long and
short, open-access papers published since 2019 that align with the rise of
Transformer-based text summarization models such as BART (?) and PEGASUS (?),
which have been the core model for most approaches in the field. This
timeframe further allows us to concentrate on current challenges rather than
including some that may have lost relevance (e.g., negation (?)) because of
new advancements in the field. We exclude non-English primary datasets, multi-
modal studies, and extractive or non-Transformer-based methods to maintain a
focus on English-language research on abstractive dialogue summarization.

Manual Paper Processing. A team of three reviewers 666Each team member
discloses potential conflicts of interest to ensure an unbiased review
process., consisting of postgraduate students and doctoral candidates with a
computer science background, undertake the manual screening process, starting
with titles and abstracts to narrow down the automatic selection from 1262 to
186 documents using our eligibility criteria. They then review the full texts,
checking if the works still meet the eligibility criteria, and decide whether
to include, exclude, or discuss each paper. Disagreements are resolved through
consensus. They also track features such as used datasets, employed backbone
models, proposed techniques, and evaluation approaches. This step filters out
53 papers, leaving 133 to compose the literature review and detail challenges,
techniques, datasets, and evaluation metrics. To further minimize bias, a
secondary review is performed as needed, reaffirming initial selections and
categorizations, thereby enhancing the study’s integrity.

##  3 Challenges and Progress

Abstractive dialogue summarization aims to condense multi-party conversations
into their key points, ranging from casual chats to expert discussions, and
whether these points were explicitly stated or inferred (?). The task was
first thoroughly defined in 2002 by ? (?), outlining challenges unique to
spoken dialogues such as handling speech disfluency, the lack of clear
sentence boundaries, distribution of salient information across various
speakers and turns, resolving references, identifying discourse structures,
and dealing with inaccuracies caused by speech recognition errors. While some
definitions take a mathematical approach (?), viewing dialogues as sequences
of turns that are compressed and each associated with a specific speaker,
others adopt a broader description of the task as ”extracting useful
information from a dialogue” (?). Although all definitions contain relevant,
overlapping aspects, they lay down different viewpoints, each considering a
different set of challenges to define dialogue summarization. Our goal is to
organize these definitions and provide a concise, structured overview of the
progress in abstractive dialogue summarization, extending the earlier
challenge-based concept (?) and proposing the Challenges of Abstractive
Dialogue Summarization (CADS) Taxonomy (Figure 1) tailored to the current
Transformer-based model architecture. This taxonomy consists of six
challenges: language, structure, comprehension, speaker, salience, and
factuality. The first five challenges are thereby related to the input, while
the last one concerns reliability. In the following Sections 3.1, 3.2, 3.3,
3.4, 3.5 and 3.6, we detail the CADS Taxonomy as shown in Figure 1.

For this section, we consider 91 papers in total, from which 73 papers define
the challenge characteristics (sub-challenges in italic) and 74 describe
explored techniques. We exclude papers mainly discussing evaluation strategies
and datasets from this subset and cover them in Sections 4 and 5.

###  3.1 Language

Characteristics. The language challenge describes the idiosyncratic nature of
spoken language and individual speech patterns. This includes linguistic
subtleties such as informal expressions (e.g., ’yeah,’), ungrammatical
structures, colloquialisms, personal vocabulary, and linguistic noise (e.g.,
filler words) (?, ?, ?, ?, ?). It further covers content repetition (i.e.,
speakers restate or rephrase information for emphasis or clarity) (?, ?, ?),
and domain-specific terminology (e.g., medical terms) (?, ?, ?, ?). These
elements demand that models accurately interpret and adapt to the linguistic
features of dialogues (?). Figure 3 presents a dialogue snippet showing
examples of these three sub-challenges. Failing to handle this challenge may
result in a loss of coherence and clarity, repetition of content, and
factuality issues.

Figure 3: Example dialogue to demonstrate examples of the idiosyncratic nature
of spoken language and individual speech patterns. Red displays disfluencies,
blue highlights personal speech patterns, orange displays the colloquialism,
green represent informal expressions

Approaches. Transformer-based models often struggle with the language
characteristics due to a gap between their pre-training data, typically well-
edited texts (e.g., news articles, research papers, Wikipedia entries) (?),
and the characteristics of spoken dialogue not reflected in these edited texts
(?). The transfer between these text styles is hindered by the scarcity of
diverse, dialogue-oriented pre-training data, resulting in a lack of exposure
to dialogue data during training, consequently reducing models’ performance.
Repetition of already provided information is less of an issue for current
language models (?).

Current research aims to bridge this gap between the formal language of the
pre-training data and dialogues by retraining models with dialogue-focused
tasks, considering both single- and multi-task setups. Explored tasks span
masking key dialogue elements (e.g., pronouns entities, high-content tokens,
and words) (?) and part-of-speech tagging (?). More recent tasks aim to
transform informal, first-person dialogue into a structured, third-person
narrative through changing speaker names, adjusting grammar, and adding
emotional context (?), or simulating a dialogue by modifying pre-training
documents into a conversation structure (?). Teaching models to understand the
’who-did-what’ structure through a specific task is further gaining traction
(?). Other explored strategies include pre-processing the input for anaphora
resolution (?) and adapted pre-training (?, ?, ?).

###  3.2 Structure

Characteristics. Aiming at extracting the built-up of a conversation, the
structure challenge revolves around segmenting a dialogue transcript and
creating an ordered summary based on the extracted structures (?). This
challenge involves identifying the topic flow within a conversation (?, ?, ?,
?), tracking the dialogue phases (e.g., problem identification, decision
making) (?, ?)), and analyzing the utterance interaction (i.e., relationship
and dependencies between utterances) (?, ?, ?). In Figure 4, we show a
dialogue snippet exhibiting topic flow, dialogue phases, and utterance
interactions. This can result in summaries lacking coherence, completeness,
and depth (?, ?, ?, ?, ?).

Figure 4: Excerpt from a conversation to demonstrate the structural challenge,
with a detailed view on one topic out of multiple topics touched in the whole
conversation. The red arcs mark the connection between individual turns,
demonstrating how different dialogue phases (grey text), such as the
conclusion, relate to many previous conversation phases.

Approaches. Transformer-based models face two primary hurdles when handling
the structural challenge, i.e., adapting to the variability in dialogue
structures, such as differing topic sequences and conversation phases
depending on the conversation type, and managing long-range dependencies to
link utterances. The first difficulty is related to the lack of varied
datasets across different dialogue types on which models can train to improve
generalization as well as that most pre-trained models used are expecting
clearer structural signals due to the more structured data used during pre-
training (Section 3.1). The second problem relates to a weakness of
transformed-based architectures: their limitation in handling long sequences,
which stems from the practical limitation of a fix-sized context window (?),
the diluted attention on a single token when many tokens are processed (?),
and the bias towards recent information (?).

Recent methods employ graph structures to understand and use dialogue
structures more effectively. They combine static and dynamic graphs for a
detailed examination of conversation dynamics (?) where the static graphs
represent unchanging aspects like speaker relationships, and dynamic graphs
track how dialogues evolve. Abstract Meaning Representation (AMR) graphs are
employed to capture overarching themes (?), detailed sentence-level
connections, and entity interactions, enhancing content comprehension (?).
Meanwhile, a hybrid approach combining traditional language models with graph
neural networks captures the utterance interactions and addresses the long-
range dependency sub-challenge (?). Similarly, ConceptNet (?) is used for
thematic linking (?).

To manage long dialogue transcripts beyond context-size limits and the
inherent long-distance relations, researchers are moving away from simply
truncating inputs to context length to avoid losing vital information. A
leading method segments the summarization process into stages (e.g., topics
and dialogue phases) for coherent processing (?, ?, ?). The segmentation and
multi-pass strategy iteratively summarizes parts of the dialogue and combines
partial summaries with a subset of the remaining content in each iteration to
build a complete summary (?). The $Summ^{N}$ framework (?), summarizes
dialogue segments independently before refining them together, offering a
detailed and scalable solution.

Other techniques involve additional training tasks (?, ?, ?, ?), sentence
importance measures to group sentences into sub-topics (?, ?), while
hierarchical setups to differentiate dialogue parts based on who’s speaking
(word level) and what’s being discussed (turn level) are also popular (?).

###  3.3 Comprehension

Characteristics. The comprehension challenge involves accurately understanding
and contextualizing utterances so models can grasp and leverage these insights
for summarization (?). This includes direct and implied content, e.g.,
unmentioned background knowledge, organizational knowledge, and prior
discussions not explicitly referred to (examples in Figure 5). To infer and
understand this content, the whole context of a dialogue is required, spanning
both the local context, focusing on neighboring words and sentences, and the
global context, covering the entire dialogue’s informational flow (?).

Figure 5: Example conversation to demonstrate the comprehension challenge
including direct and implied content. Directly stated (orange background) are
the explicit updates by Bob and Charlie about their respective tasks. Green
marks implied content such as a deadline for a demo and a current focus on
performance and finalizing features (unmentioned background knowledge), Redis
implies an understanding of its role in performance enhancement
(organizational knowledge and reference to prior discussions on data
requirements).

Approaches. Transformer-based models struggle with comprehension and
contextualization due to the underlying dependency parsing challenges,
particularly with unclear sentence boundaries as they appear in spoken
language, which can lead to catastrophic forgetting and result in instability
in handling long-range dependencies (?). Additionally, these models often
interpret the text literally, affecting their ability to grasp nuanced
meanings (?, ?). As these models often rely on syntactic relationships between
words, they miss the nuanced understanding humans use to interpret implied
meanings of sarcasm, irony, euphemisms, and similar pragmatic elements (?).
Consequently, this can lead to summaries that miss deeper meanings or
implications, resulting in content that may be shallow or misleading (?).

Despite its importance, our review reveals limited engagement from the
dialogue summarization community to improve the contextualizing capability of
models considering both the short and long context. Current efforts primarily
rely on Transformer models’ self-attention capabilities to comprehend local
and global contexts (?). Despite this reliance, the context-aware extract-
generate framework (?) has been proposed to reduce the risk of missing long-
range contexts in complex dialogues, which uses context prompts to capture the
immediate details and the overall narrative of text spans. It first pinpoints
relevant text parts that guide the extraction of key information, which is
then used as a prompt base for generating detailed summaries.

###  3.4 Speaker

Characteristics. Participants, their interactions, and how they influence the
dynamics of a conversation are part of the speaker challenge. This includes
the individual participant roles (e.g., ”project manager”, ”customer service
agent”) (?, ?), topic-dependent role changes (?, ?, ?, ?, ?, ?, ?), their
interactions (e.g., contributions depending on viewpoints (?, ?, ?),
backgrounds (?, ?, ?)), and references to entities and other participants (?,
?, ?, ?). Examples of the speaker characteristics are shown in Figure 6.
Failing to capture these speaker characteristics may lead to missing context,
misaligned entity associations, and, ultimately, incorrect factual summaries.
The significance of examining speaker dynamics and roles is well-recognized in
other fields such as linguistics (?) and is, together with the language
challenge of the preceded analysis, a common hook to demonstrate and motivate
techniques for dialogue summarization.

Figure 6: Example conversation of the speaker challenge with defined roles
(Alex as customer, Jordan as customer support agent, Taylor as technical
specialist). The dialogue shows a change of roles when Jordan calls Taylor
(marked with orange), shifting from Jordan as a primary troubleshooter to a
facilitator for Taylor’s technical interventions. Further, multiple direct
references to entities highlighted with blue (e.g., ”device”, ”error codes”)
and indirect references to participants in red (e.g., ”you”) are incorporated.

Approaches. The speaker challenge arises from identifying participants,
tracking their actions and entity references, and revealing participant
relationships. These challenges can be aligned with the difficulties of named
entity recognition (NER), coreference resolution, and dependency parsing.
Although advancements with Transformer-based models regarding NER, the
explored techniques struggle to accurately identify and categorize entities
across different text types and domains and with unstructured texts (?, ?, ?),
rendering realistic dialogue scenarios as a challenging field. Coreference
resolution poses difficulties for techniques in accurately identifying
mentions and contextual links (?) and understanding the linguistic structures
and relationships (?), extending the understanding challenge (Section 3.3).

Works typically use dense vectors (?, ?) to accurately represent speaker roles
in dialogue and act as labels for utterances (?, ?), offering insights into
each speaker’s function within the dialogue. Speaker vectors are either
randomly initialized and trainable (?, ?) or produced by small neural networks
(?, ?) and represent different roles, e.g., ”industrial designer” (?), ”judge”
(?), which are appended to the embedding of the speaker’s turn. For capturing
speaker dynamics and resolving coreferences, graph-based models are an
established approach, e.g., to build a graph representing each speaker’s main
ideas and their discourse, alongside their interactions, reflecting inner- and
inter-speaker structures (?, ?).

Some other popular techniques involve enhancing Transformers’ self-attention
to analyze intra- and inter-speaker dynamics (?, ?), additional training tasks
(?, ?, ?), and pre- and post-processing techniques to replace pronouns and
correct mistakes (?, ?, ?, ?) for coreference resolution.

###  3.5 Salience

Characteristics. The salience challenge involves identifying and summarizing
the most relevant information within a dialogue (?). This includes identifying
content critical for insightful summaries (?), where the understanding of
critical shifted lately from content for general audience summaries to
summaries aligning with the viewpoints, values, and ethics of specific
participant (?), and locating text spans containing relevant content (?).
Figure 7 demonstrates the differences between personalized and generally
applicable summaries, as the generated texts for Simon and Nadia are tailored
to their role, stands, and tasks, whereas the general summary provides a
neutral overview. Failure to effectively locate the scattered information or
understand what is considered salient can impact the depth and relevance of
the generated summaries, focusing on unimportant content while omitting
relevant ones (?, ?, ?, ?).

Figure 7: Example of the salience challenge with a conversation being
summarized for a general audience and tailored summaries for participants with
a specific focus on content relevant to them.

Approaches. The first hurdle for current models to the salience challenge is
the request for subjective salience and ranking content’s importance to align
with the role and knowledge of the summary addressee. This is difficult as
there is no training task for inferring personal attributes and ranking
information according to them. Further, locating this content is difficult as
salient information is typically scattered across multiple turns within the
idiosyncrasies of spoken language (?, ?, ?), which forms an extension of the
language challenge (Section 3.1).

To nest the general understanding of what is considered salient into models,
established strategies include additional training stages to better
distinguish between salient and non-salient content. ? (?) introduce the
uncovered loss to point out if salient information is missing and the
contrastive loss to distinguish between relevant and irrelevant sentences,
respectively. Negative samples, categorized as redundant (i.e., text with
unnecessary utterances) and null (i.e., text with relevant utterances
removed), help models prioritize important content (?, ?). To better determine
salient content for specific participants, works propose a fine-tuning task in
which the topic or user perspective to be focused on is passed as an input
along the dialogue text (?), apply question-driven pre-training (?) and use
dynamic prompts to direct model attention to key dialogue aspects (?).
Additional strategies involve direct text manipulation (?), personal named
entity planning to concentrate on specific entities (?), adapting attention
mechanisms (?), and integrating human feedback (?) or graph structures (?) to
further tailor summaries to the dialogue’s essential parts.

###  3.6 Factuality

In contrast to the previous input-related challenges, the factuality challenge
describes the problem of correcting false content in a generated summary
caused by a non-robust architecture. Factual incorrect content contains
intrinsic errors, such as hallucinating events not present in the original
transcript, and extrinsic errors, such as incorrect coreference resolution
misrepresenting actual event details, and wrong conclusions stemming from
negation (?). Intrinsic and extrinsic errors are shown in a sample dialogue in
Figure 8. Given the importance of ensuring dialogue summarizers accurately
reflect the conversation, research focuses on creating safety measures to
maintain summary credibility despite these challenges.

Figure 8: Example of the factuality challenge where the generated summary
exhibits intrinsic errors through hallucinating events (blue background) such
as the finalization of a marketing strategy and a detailed review of financial
strategies. The extrinsic error reporting the wrong date of the meeting is
marked with red.

Approaches. Research interest predominantly targets post-processing approaches
that rewrite the generated summary to correct factual errors. Notable
approaches here are the usage of LLMs for error detection and correcting (?)
and reformulating the summary to closely match the original dialogue’s
predictive capabilities for subsequent content (?). Other approaches explore
the use of auxiliary tasks (?) to estimate the factual aspects and predict the
missing aspects in a summary (?), losses such as encouraging a model to
generate sentences about content not yet covered and differentiating factual
from non-factual sentences (?, ?), and human feedback (?). Further,
architecture-modifying approaches are explored, such as a new encoder to grasp
dialogue states (?) or to handle graph structures, where events of dialogue
are captured and organized in a graph. A special attention mechanism is used
to fill in summary (?), and a hierarchical approach builds on sub-summaries
(?). Another noteworthy technique is the generation of a sketch to structure
out the final summary along this plan (?). Negation is a long-standing
challenge for language models often neglected in computational studies (?, ?).

##  4 Datasets

In this review, we collected both methods and downstream datasets from the
selected publications. We only consider datasets used at least five times in
our analysis to avoid including resources with little community impact. This
yields 18 datasets that we group into six categories based on the dialogue
subdomains they stem from: daily chat, online chat, meeting, screenplay,
customer service, and medical. We also create another category for datasets
that don’t fit neatly into these subdomain groups.

These datasets typically condense a short dialogue of about 1k tokens between
two real persons into a third-person summary of about 100 tokens. More complex
scenarios, such as meetings, parliamentary debates, and TV series dialogues,
involve around 10k to 20k tokens in the transcript and five or more
participants. Due to privacy concerns, a significant portion of currently
publicly available datasets are reenactments based on real events, e.g., by
Amazon Mechanical Turk workers, or established datasets such as SAMSum (?) and
AMI (?) are modified to generate new data, e.g., by changing the dialogue
transcript to a third-person report (?). This underscores that the variety of
existing datasets is comparably limited, and these datasets may not capture
real-life scenarios. The 18 available established datasets are detailed in
Sections 4.1, 4.2, 4.3, 4.5, 4.7, 4.6, 4.4 and 4.8. Given the limited number
of available datasets, we present methods for creating artificial datasets and
strategies for optimizing the use of existing datasets in Section 4.9.

For this section, we consider 93 papers to identify the established datasets,
excluding works that do not mention a dataset, datasets that are no longer
publicly available, and datasets in a language other than English. We did not
assess dataset quality or perform a detailed analysis of the inherent
challenges, leaving that for future research. In Figure 9, we connect the
datasets with their respective primary challenge, as noted by their creators,
to provide an overview of the distribution of challenges in datasets.

Figure 9: Matching between established datasets and their primary reported
challenge.

###  4.1 Daily Chat

DialogSum (?) integrates dialogues from DailyDialog (?), DREAM (?), MuTual
(?), and English-speaking practice websites, featuring two-speaker
interactions across daily-life scenarios like work, leisure, and travel.
Annotators were given guidelines on writing the summaries, such as length
constraints (no longer than 20% of conversation length), maintaining an
observer perspective, and explicitly describing important emotions related to
events in the summary. DialogSum includes 13k dialogues, with inputs of around
1k tokens and summaries of 130 tokens.

###  4.2 Online Chat

SAMSum (?) comprises 16k written online dialogues from messaging apps, which
linguists craft asked to create conversations similar to those they write
daily, reflecting the proportion of topics of their real-life messenger
conversations. The messages, each written by one person, contain chit-chats,
gossip, arranging meetings, discussing politics, and consulting university
assignments. They typically involve two speakers, with an average of 94 tokens
per conversation, varying between three and 30 turns 777A turn is the
contribution made by a speaker in the form of a single utterance or a
statement.. A subset of SAMSum has been adapted into formal third-person
language (?) to help models transition from informal dialogue to edited text,
addressing the linguistic gap between pre-training and downstream task
(detailed in the language challenge, Section 3.1).

FORUM (?) consists of random samples of 100 threads from the online discussion
forums on Ubuntu and TripAdvisor, with a total of 556 and 916 posts,
respectively. Two human evaluators created summaries of the discussion
threads, resulting in two human written summaries per sample.

CRD3 (?) consists of dialogues from the ”Critical Role Dungeon and Dragon”
show with summaries collected from the Fandom wiki, featuring 159 episodes
with an average dialogue length of 2550 turns.

###  4.3 Meeting

AMI (?) contains business meetings on the product design process, detailing
137 staged meetings on designing a new remote control. Participants are
project managers, marketing experts, user interface, and industrial designers.
The dataset includes transcripts and human summaries, with dialogues averaging
6k tokens over 535 turns and four speakers. A modified version, AMI-ITS (?),
offers additional annotations for incremental temporal summaries, which
provide summaries for 100-second time durations on a subset of AMI.

ICSI (?) has 59 academic group meetings with computer scientists, linguists,
and engineers at the International Computer Science Institute (ICSI) in
Berkeley, along with their summaries written by hired annotators. The meetings
have an average of 819 turns and 13k tokens with six speakers and are about
research discussions among students.

QMSum (?) introduces query-based summarization across diverse meeting domains,
compiling 1.8k query-summary pairs from 232 meetings, encompassing product
design (AMI), academic discussions (ICSI), and committee deliberations. The
dialogues feature up to 13k tokens and six speakers. The original task is to
summarize the meeting given a stated question. MACSum-Dial (?) is a
modification of QMSum, designed for controllable summarization, highlighting
mixed attributes such as length, attractiveness, and topic specificity.

ELITR (?) features 120 English technical project meetings in computer science,
with each transcript averaging 7k words, 730 turns, and six speakers. The
duration of the meetings varies from ten minutes to more than two hours, with
an average of one hour long. Meetings shorter than half an hour are
exceptions, whereas meetings longer than two hours are topic-oriented mini-
workshops, also rather occasional. The original task here slightly differs as
a model is required to produce not an abstractive summary but a set of meeting
minutes in bullet points.

MeetingBank (?) contains meetings of city councils from six major U.S. cities
occurring over the past decade. It contains 1366 meetings spanning 3.6k hours,
with a council meeting lasting an average of 2.6 hours and the transcript
containing 28k tokens.

###  4.4 Screenplay

MediaSum (?) contains 463.6k media interview transcripts with brief summaries
from National Public Radio (NPR) (?) and CNN, spanning a range of topics. The
summaries are based on NPR’s interview overviews and CNN’s topic descriptions,
with the latter segmented at commercial breaks to match topics to
corresponding interview segments. On average, each transcript in this dataset
contains 1.5k words, with summaries of around 11 words, typically involving 7
speakers.

SummScreen (?) consists of 27k instances of TV series transcripts paired with
human-written recaps sourced from TV MegaSite and ForeverDreaming. The recaps
of ForeverDreaming are based on community contributions stemming from
Wikipedia and TVMaze. Transcripts, typically including 28 speakers, average
6.6k tokens, and summaries around 380 tokens.

###  4.5 Customer Service

TweetSumm (?) contains 1.1k dialogues derived from Twitter customer support
exchanges, each with three extractive and three abstractive human-written
summaries. Originating from the Kaggle Customer Support On Twitter dataset,
these real-world interactions span various industries, such as airlines and
retail, averaging 10 turns per dialogue and 36 tokens per abstractive summary.

TODSum (?) is a customer service dataset based on MultiWOZ (?), from which
they select the five domains: restaurant, hotel, attraction, taxi, and train.
The dataset spans 10k samples with an average dialogue length of 187
utterances and 45 words in the summary.

###  4.6 Medical

MTS-Dialog (?) is a collection of 1.7k simulated doctor-patient conversations
with corresponding clinical notes serving as summaries, sourced from the
public Mtsamples collection (?). This dataset encompasses various medical
fields, including general medicine, neurology, orthopedics, dermatology, and
allergy/immunology, adhering to the SOAP (Subjective, Objective, Assessment,
Plan) note format. Dialogues average 18 sentences and 242 words, with
summaries typically around 81 words.

###  4.7 Debates

ADSC (?) features sequences of two-party dialogue chains derived from the
Internet Argument Corpus (?), focusing on significant social and political
topics like gun control, gay marriage, the death penalty, and abortion. It
includes 45 dialogues, each accompanied by 5 unique human-generated summaries,
with each summary approximately 150 words in length.

###  4.8 Others

ConvoSumm (?) consolidates dialogues from four sources: New York Times
comments, StackExchange, W3C emails, and Reddit, totaling 2k dialogues with
500 from each domain. Crowdsourced workers from Amazon Mechanical Turk wrote
the abstractive summaries with at most 90 tokens. Inputs average 1.1k tokens,
with summaries about 70 tokens in length.

FERRANTI (?) is a dataset designed for factual error correction in dialogue
summarization, featuring 4k manually annotated items. The original task is to
evaluate the factuality of summaries and how to correct these summaries with
minimal effort. Drawing on SAMSum and DialogSum, FERRANTI includes summaries
produced by models like BART (?) and UniLM (?). Human annotators assess these
summaries for accuracy and identify errors, providing a focused tool for
improving summary factualness.

DialSum (?) is based on the VisDial dataset (?), where two participants
discuss an image from the MSCOCO dataset (?), which features approximately
120k images. Each image thereby has five captions from five different
annotators.

###  4.9 Data Augmentation and Utility

Besides highlighting datasets with a notable community interest, we summarize
in this subsection the research on techniques to cope with the data scarcity
in dialogue summarization, covering artificial generation techniques of
datasets through text generation and curriculum learning strategies to use the
available data more effectively.

Artificial Dataset Generation. As discussed earlier, dialogue summarization
faces challenges due to the need for adaptability across various domains,
structures, and speaker dynamics, which typically would be addressed by
training models on diverse datasets (?). However, the scarcity and small size
of datasets, with utterances sometimes containing just two to ten turns in
areas like customer service and medical, restrict model training for more
general applications.

Creating real-world datasets is costly and may conflict with data security and
information communication policies, so artificially generated datasets are
explored (?). Methods range from simple heuristic-based weak labeling, such as
selecting the leading or longest utterance as proxy summary (?), to
paraphrasing with updating the summary to maintain coherence (?, ?, ?), to
random alterations (e.g., swapping, deletion, dialogue-act-guided insertions)
and changing conversation structures (?, ?). The most common method is to use
language models to create ground truth summaries either directly (?, ?, ?, ?)
or after training them on human summarization techniques through few-shot
learning (?).

Techniques to Maximize Dataset Utility. As training Transformer-based models
requires extensive data to generalize across various topics and dialogue
formats (?), researchers have explored ways to use the limited data available
more effectively. A key strategy involves a prompt-based curriculum learning
strategy that gradually increases the degree of prompt perturbation (e.g.,
word swapping, content cutting) to improve the generalization ability of
models (?). Another variation is dynamic prompts, which select best-fitting
few-shot samples considering dialogue content, size, and speaker number (?).
Also explored are prompt transfer techniques from related dialogue domains (?)
to bolster the usage of dialogue state information. Prompts are further used
to split inputs into domain-invariant and domain-specific content to enhance
model generalization (?, ?). Other relevant techniques include freezing most
model parameters and training only specific parameters for domain adaption (?,
?, ?).

##  5 Evaluation

Researchers have adopted metrics from related fields and introduced new ones
designed to assess the effectiveness of dialogue summarization methods. These
metrics are used as a proxy for quality characteristics, such as coherence,
fluency, factuality, and accuracy. In our literature review, we identify 15
metrics used more than twice in 93 papers. We categorize these metrics into
four groups: count-based (e.g., ROUGE (?)), model-based (e.g., BARTScore (?)),
QA-based (e.g., QAEval (?)), and human evaluation. The automatic metrics
strive for alignment with human judgment. However, the correlation between
these two is weak for dialogue summarization, and automatic metrics sometimes
reward low-quality texts (?). Due to these recognized limitations in fully
capturing the nuances of summarization quality, human evaluation is considered
the gold standard across most studies. Recent analyses (?, ?) conclude that
established automatic metrics may work for a superficial understanding of a
model’s performance but do not align well with human judgment in discerning
error nuances. These metrics either do not adequately display error occurrence
or cannot reflect the impact on the quality of their scores. Hence, a
composite metric of count-based, model-based, and QA-based metrics may be
required for a more reliable automatic interpretation.

Section 5.1 details the characteristics typically evaluated by the metrics and
Sections 5.2, 5.3, 5.4 and 5.5 further summarizes the 15 established metrics
in dialogue summarization. Table 1 provides an overview of the identified
metrics and their usage throughout the papers considered for this literature
review.

###  5.1 Assessed Characteristics

Metrics assess the quality of a generated text, often in relation to a
reference text. Low-quality text may contain more redundancy, incoherence,
grammatical errors, poor structure, or inappropriate language, while high-
quality text would closely align with the reference. We identify ten key
characteristics (in italic) organized into four overarching groups: accuracy,
content, readability, and style.

Accuracy is the core of a high-quality summary (?), making adherence to the
original text’s truth and, therefore, factuality paramount. A summary must
accurately reflect the facts, events, and details from the source without
distortion.

Content quality is mainly driven by relevance. A summary must contain the most
relevant information, directly addressing the addressee’s informational needs
(?). This effort to overcome the salience challenge ensures that summaries
prioritize the most important points for the reader. Coverage complements
relevance by ensuring that all key topics and arguments from the dialogue are
included, offering a comprehensive understanding without significant omissions
(?). The goal is to track both the salience and structure challenges with
this. Informativeness goes a step further by selecting crucial information
that conveys the depth of the input transcript, enabling readers to grasp the
main points without needing to refer back to the original text (?).

Readability is essential for making summaries accessible and easy to
comprehend. Coherence ensures that information is presented logically with
smooth transitions, making the summary easy to follow (?). The structure and
organization of a summary further enhances its readability and facilitates
information processing, ensuring that it is well-organized and logically
constructed (?). Conciseness (?) and non-redundancy (?) are also essential as
a summary should contain only the essence of the input transcript without
unnecessary details.

Style further contributes to the perceived quality of a text. Consistency in
perspective, tense, and style throughout the summary contributes to its
readability and professional quality (?). Fluency focuses on grammar,
vocabulary, and sentence structure and aims for a natural, easy-to-read
summary that is free from phrasing errors, enhancing the information’s overall
clarity and accessibility (?).

###  5.2 Count-Based

Count-based metrics, including N-gram-based measures like BLEU (?), and ROUGE
(?), and statistical measures such as Perplexity (?), are often static and
rule-based algorithms that have a long history for evaluating text
summarization. However, due to their simple approach, they have been
criticized for their limitations in capturing overall meaning, fluency,
coherence, or factuality in their scores (?).

BLEU (?) measures the precision of generated summaries against reference texts
by examining the overlap of word sequences (typically 1- to 3-word N-grams
(?)), emphasizing similarity to the original phrasing but risking a linear
relationship to noise (?).

ROUGE (?) focuses on lexical similarity (?) and aims to capture the extent to
which key content from the source is included in the summary, prioritizing
content coverage through recall of N-gram overlap, thus attempting to address
BLEU’s limitations by emphasizing content inclusion over mere precision. A
frequently used variation is ROUGE-L, which measures the longest common
subsequence between a candidate and a reference.

METEOR (?) advances BLEU and ROUGE by incorporating both precision and recall,
along with synonym matching for semantic analysis, thus enabling to capture
the semantic similarity between a candidate and a reference. Despite its more
balanced approach and sentence-level focus, METEOR, like its predecessors, is
prone to noise interference (?).

chrF++ (?) further expands the previous metrics by considering precision,
recall, and F-score-based N-gram overlap at both character and word levels,
offering a more granular analysis (?).

CIDEr (?) is a consensus-based metric that compares the similarity of a
generated sentence against a set of human-written reference sentences. The
score is an aggregation of cosine similarity scores between the TF-IDF
weighted N-grams of the generated and reference sentences, inherently
capturing precision, recall, grammaticality, and salience (?, ?, ?).

Perplexity (?) diverges from the previous methods by the statistical approach
of gauging a model’s uncertainty in predicting word sequences, with lower
scores indicating better alignment with expected language patterns. This
measure is, in contrast to the previous, an intrinsic evaluation metric that
directly evaluates the language modeling objectives through text
predictability rather than assessing the performance on the downstream task.
Therefore, additional metrics are required for a holistic evaluation of text
generation quality.

###  5.3 Model-Based

With advancements in language models, there is an increasing focus on model-
based evaluation metrics due to their higher correlation with human judgment,
though count-based metrics remain popular. Model-based metrics range from
those using handcrafted features and learned embeddings to hybrids combining
contextual embeddings with count-based approaches. After representing the
respective text in a latent space, these metrics consider the similarity
between the candidate and reference text, aiming to offer a more nuanced
assessment by focusing on semantic similarity, likelihood, and usefulness of
summaries. However, despite their potential to adapt to changes in language
use (e.g., distribution drifts) (?), these metrics can still be error-prone
(?), slower than count-based metrics and may not directly measure the specific
characteristics outlined in Section 5.1, making it challenging to discern
which particular aspect influences their scores.

BERTScore (?) leverages BERT embeddings (?) to contextually embed the
reference and candidate text, calculate the pairwise cosine-similarity on the
token level to build a similarity matrix, and greedily pick the highest
similarity scores to compute the harmonic mean of precision and recall,
thereby capturing semantic nuances (?).

MoverScore (?) was introduced concurrently to BERTScore following a similar
approach, as it also leverages BERT embeddings (?) but instead considers the
distance between reference and candidate text, hence employing the Word
Mover’s Distance (?), a special case of the Earth Mover’s Distance (?), to
measure semantic distance. This change in measure allows MoverScore to map
semantically related words from one sequence to one word in the other sequence
(many-to-one).

BARTScore (?) evaluates the plausibility of generating a reference text from a
given generated text, and vice versa, by calculating the log-likelihood of a
sequence that a BART (?) model would typically generate based on the given
context. This evaluation focuses on assessing both the fluency and the
semantic accuracy of the generated content.

BLEURT (?) extends beyond mere embedding comparisons by incorporating a BERT
model pre-trained on lexical- and semantic-level supervision signals and fine-
tuned on human judgments, enabling it to make detailed assessments of text
quality, including coherence and relevance.

BLANC (?) leverages BERT to perform a fill-in-the-blank task, both with and
without the generated summary, to assess how informative or helpful the
generated text is. The difference in prediction accuracy indicates the utility
of the summary in helping understand the text.

FactCC (?) evaluates a summary’s factual consistency with its source document
using an entailment classifier, scoring based on the proportion of sentences
classified as factual consistent by a BERT model. The metric struggles with
complex inferences and subtle nuances beyond direct comparison.

###  5.4 QA-Based

The QA-based metrics we identify in the literature concentrate on the factual
correctness of a summary using external question-answering systems. The
effectiveness of these metrics largely depends on the performance of the QA
systems, which may not always align with human evaluation standards.

FEQA (?) generates questions based on the summary content and verifies whether
their answers can be found in the source document.

SummaQA (?) derives questions from the source text and tries to answer these
using the summary. It expands the QA-based evaluation scope by incorporating
various question types and emphasizing the summary’s informativeness.

QuestEval (?) combines the approaches of FEQA and SummQA, thereby adopting a
bidirectional strategy, generating questions from the summary to compare with
the source text and vice versa. This bidirectional evaluation offers a
balanced and holistic assessment by considering included and omitted
information in the summary.

Type |  Category |  Metric |  Descriptive Tags | Usage
---|---|---|---|---
Count-based |  N-gram |  BLEU |  word overlap, precision, multiple references | 21
|  |  ROUGE |  word overlap, recall, one reference | 127
|  |  METEOR |  word overlap, precision and recall, one reference | 3
|  |  chrF++ |  character-level, F1 score, one reference | 2
|  |  CIDEr |  consensus-based, multiple references | 3
|  Statistical |  Perplexity |  likelihood of word sequences | 3
Model-based |  Hybrid |  BERTScore |  token similarity, cosine-similarity, one-to-one | 38
|  |  MoverScore |  token similarity, mover distance, one-to-many | 3
|  |  FactCC |  entailment classifier, scoring based on consistency | 9
|  Trained |  BLEURT |  mimics human judgement | 7
|  |  BARTScore |  mimics human judgement, promptable | 13
|  |  BLANC |  fill-in-the-blanc task with and without the summary | 2
QA-based |  |  FEQA |  questions based on summary, answered through input | 38
|  |  SummaQA |  questions based on input, answered through summary | 3
|  |  QuestEval |  combination of FEQA and SummQA | 3
Human Evaluation |  Performance |  Likert Scale |  scale, typically 1 (worst) to 5 (best) | 6
|  |  Pairwise Comparison |  pick best sample out of two | 9
|  |  Best-Worst Scaling |  rank a list of samples | 5
|  Agreement |  Krippendorff’s Alpha |  measure disagreement, different data formats | 2
|  |  Cohen’s Kappa |  measure agreements, categorical data, 2 raters | 3
|  |  Fleiss’ Kappa |  measure agreement, nominal data, 2 raters | 2
Table 1: Relevant metrics and evaluation measures employed in dialogue
summarization with more than twice reported use. ’Usage’ states the number of
papers reporting the evaluation measure in their publication.

###  5.5 Human Evaluation

Human evaluation is often considered the gold standard for assessing the
quality of a summary. It is typically performed through crowdsourcing
annotators (e.g., Amazon Mechanical Turk) where they label samples. We
identify established approaches for human evaluations on summary performance
and annotator agreement.

Performance. Summary performance is typically evaluated using Likert scales
(?, ?, ?, ?), which provide a simple rating system for quality assessment but
lack detailed feedback on specific text issues (?). Alternatives like pairwise
comparison (?, ?) and best-worst scaling (?, ?) offer more nuanced
evaluations, with best-worst scaling noted for its higher reliability (?).
Despite these options, the Likert scale remains the predominant method.

Agreement. The reliability of human-generated evaluations hinges on annotator
agreement, also referred to as meta-evaluation (?). Despite its significance,
the incorporation of agreement metrics is frequently overlooked. We identify
three established measures to determine inner-annotator agreements:
Krippendorff’s Alpha, Cohen’s Kappa, and Fleiss’ Kappa. Their scores range
from 0 (poor reliability) to 1 (perfect reliability), with reported scores
typically between 0.65 and 0.85. Krippendorff’s Alpha (?) offers a flexible
approach suitable for multiple raters, applicable to any level of measurement,
i.e., nominal, ordinal, interval, or ratio, and accommodates both qualitative
and quantitative assessments. This measure is particularly valuable with a
broad range of variables or when comparing agreements across different
measurement scales but assumes that all annotators assess all samples. Cohen’s
Kappa (?) is used to measure the agreement between two raters who each
classify a set of items into mutually exclusive categories (e.g., yes/no).
Fleiss’ Kappa (?) is also tailored for scenarios involving fixed-category
classifications but extends Cohen’s Kappa to multiple annotators, excelling in
the evaluation of how consistently annotators categorize text segments or
dialogue turns into predefined groups.

##  6 Discussion

###  6.1 Remarks on Challenges

In this study, we have introduced the CADS taxonomy to organize the inherent
challenges of abstractive dialogue summarization (i.e., language, speaker,
salience, comprehension, structure, and factuality) and unify their underlying
definitions. While presented separately for clarity, the challenges are
interdependent and influence each other. Consequently, approaches should be
researched to tackle the challenges simultaneously whenever possible. Despite
advancements in addressing the challenges, we observe that most of them are
still a hurdle for models due to limitations in the Transformer components
used in these models, the lack of capabilities in contextualization and few-
shot adaption of the encoder-decoder backbone models, and missing mitigation
techniques.

Since 2023, NLP has seen a significant shift with the exploration of LLMs and
optimized Transformer architecture components (e.g., ring attention (?) and
multi-token generation (?)), with their application to dialogue summarization
being explored later (?, ?, ?). Initial studies indicate that LLMs match or
exceed the performance of task-specific models like DialogLED (?), even with
their limitations (e.g., hallucinations, salience) (?). ? (?) show that
encoder-decoder models may perform better against the speaker challenge,
particularly regarding coreference resolution, and struggle with the structure
challenge. LLMs handle the comprehension challenge but seem to be sub-optimal
regarding robustness to language and speakers. These early observations state
LLMs as a noteworthy alternative to more traditional techniques (e.g.,
dialogue-style pre-training (?), graph structures to represent speaker
structures (?), and special losses tailored to determine salience (?)).

In the following, we discuss how the hurdles of the currently employed
encoder-decoder models (identified in Section 3) align with challenges in NLP
and discuss techniques introduced so far to mitigate these challenges, thereby
deriving how they can aid dialogue summarization.

Bridging the language gap. Adapting established summarization models from
structured pre-training data to the less formal nature of dialogue remains
challenging. LLMs, especially those trained on non-edited text such as models
from the GPT and GEMINI series, are promising in this regard (?). Their few-
shot learning capabilities allow for rapid adaptation to new linguistic
patterns more effectively. At the same time, the pre-training on a large and
diverse corpus improves robustness in handling informal language, style
variations, and repetition (?).

Long distance handling and dependency parsing. Recent progress in LLMs such as
GPT-4, Claude-3, and Phi-3 shows a significant extension of the processable
context length to over 100k tokens, exceeding the context size of earlier
models like LED (?), which handle up to 16k tokens. This increase in context
allows for handling long dependencies without chunking, which is crucial for
understanding complex dialogue structures, speaker dynamics, and content
understanding. While traditional models could theoretically handle such
lengthy contexts, practical limitations arise from the input-length-dependent
quadratic computational costs associated with Transformer attention mechanisms
(?). To address these issues, techniques like sparse attention (?), flash
attention (training stage) (?) and ring attention (inference stage) (?),
sharing weights across attention heads (?), and conditional computation to
reduce the overall computational load (?) are proposed to manage large inputs.

Generalizability. Modifying pre-trained models to handle the salience and
structure challenges independent of the conversation’s domain is essential for
practical applications (e.g., customer service). Established generalization
techniques involving prompting (?, ?), few-shot learning (?, ?), and meta-
learning (?, ?) are incipient in dialogue summarization. Meanwhile, LLMs build
on these techniques to leverage patterns observed during the training stage
and boost their generalization capabilities (?). However, LLMs may still
struggle with bias mitigation and ensuring broad applicability across various
contexts, when ”encountered words are not present in the training corpus or in
examples (?, ?, ?).

Coreference resolution. Large-scale pre-trained and fine-tuned language models
currently set the benchmark within NLP for coreference resolution (?).
Challenges remain with the ambiguities in reference and context (?), and the
usage of LLMs for this task.

Hallucination reduction. Avoiding hallucinated content in generated text is a
major challenge in NLP tasks using LLMs, especially in culturally sensitive
contexts (?, ?). Mitigation techniques include confidence penalty
regularization (?, ?) to reduce overconfidence and enhance accuracy.

These advancements throughout NLP suggest upcoming shifts in the key
challenges of dialogue summarization.

  * •

The language challenge can lose impact if few-shot learning is used to prompt
LLMs, making it easier to apply document summarization techniques to dialogues
directly without additional finetuning.

  * •

We expect progress for the structure challenge with LLMs as the backbone
model, but robustness issues may persist due to issues with generalization and
processing long texts.

  * •

The comprehension challenge, especially in grasping implied meanings, remains
largely unsolved and holds significant research potential. Research from
related NLP fields such as sentiment analysis (?) suggest that current models
still struggle to understand implied content. At the same time, research on
retrieval-augmented generation (?) could enhance contextualization, helping
models better grasp the context of conversations.

  * •

For the hurdles implied in the speaker challenge, i.e., coreference resolution
and dependency parsing, LLMs are explored but have not yet succeeded due to
robustness weaknesses (e.g., errors due to uncommon names, overseeing details
in long inputs), signaling that further research is required.

  * •

The salience challenge currently builds on personalizing summaries and
understanding long discussions, which both are gaining attention lately. Due
to the novelty, the personalization of summaries requires more research on
techniques and benchmarking datasets.

  * •

The ongoing issue of hallucination in language models continues to pose a
challenge in ensuring the factuality in generated summaries.

###  6.2 Trends in Datasets

Personalized Summarization. In contrast to the typically general summaries,
personalized summaries tailored to the users’ needs are becoming increasingly
popular (?, ?). Recent advancements have explored cross-attention and decoder
self-attention to enhance role-specific information capturing (?). By
integrating detailed personal attributes, models can better understand each
participant’s background and motivations, leading to more targeted summaries.
Dataset-wise, only a few, such as the Chinese CSDS (?), incorporate
personalized summaries, whereas established English datasets typically provide
only a single, general summary.

Realistic Datasets. As models improve in few-shot learning and do not
necessarily require finetuning, the reliance on large-scale datasets is
shrinking. Considering this trend, we expect a rise in smaller datasets, which
allow for high-quality, realistic examples. These examples may be sufficient
to adapt models to new domains, make approaches more robust, and test them
against realistic scenarios.

###  6.3 Trends in Evaluation

Evaluation metrics, vital for indicating performance and comparing new
techniques, are currently mostly borrowed from related NLP tasks such as
translation or text generation (Section 5). However, the effectiveness of the
established metrics (e.g., ROUGE for dialogue summarization as identified in
Table 1) is limited (?, ?), and while it can serve as a proxy (?) it provides
insufficient insights into the true efficacy of new techniques. The also
popular model-based metrics (e.g., BARTScore) seem unable to align well with
human judgments (?) for dialogue summarization. Recent developments in NLP use
LLM-based metrics such as GEMBA (?) and ICE (?) for evaluation (?), building
on LLMs’ advanced text comprehension. This set of metrics thereby mimics human
judgment and assesses common aspects (e.g., fluency, coverage, coherence) with
continuous (?), Likert scale (?, ?), or probability scores (?). LLM-based
metrics offer a promising direction for dialogue summarization evaluation due
to their customizability, though this area remains largely unexplored.

##  7 Final Considerations

###  7.1 Limitations of Evidence

Given our selection of papers on dialogue summarization, we found limitations
in current works that may bias our findings. First, the crawled research
studies in dialogue summarization predominantly focus on English-language
dialogues, which renders their performance in multilingual or cross-cultural
contexts unknown. While we frequently came across Chinese datasets, other
languages are underrepresented. As a consequence, we focus our work on English
dialogue summarization. Second, most datasets stem from specific sectors such
as customer service or business meetings, which may not reflect the variety of
dialogue types found in different environments. Lastly, the lack of detailed
human evaluation reporting across research works, including inter-annotator
agreement scores, further complicates the assessment of new techniques’
usefulness.

###  7.2 Limitations of Review Process

In our systematic review of dialogue summarization, we encountered
methodological challenges that we had to circumvent. First, we focus our
search on studies published in English and primarily involving English
datasets. While this is a limiting factor, most research on the dialogue
summarization task considers only English dialogues, making this a feasible
approximation. However, we encourage others to assess the same issues we point
out for other languages. Second, despite considering multiple databases, the
timing and methods of our search may have missed studies, thus not fully
capturing the dialogue summarization field. To mitigate this limitation, we
updated our crawl several times throughout writing this article and retrieved
works from two databases. We considered both peer-reviewed and non-peer-
reviewed papers which risks including less credible studies or misinformation.
To mitigate the potential weaknesses of non-peer-reviewed works, we use the
ranking from semantic scholar and DBLP, which should account for the paper’s
quality and evaluate the studies’ quality based on adherence to established
reporting standards of top-tier conferences.

###  7.3 Conclusion

In this article, we reviewed works on Transformer-based abstractive dialogue
summarization. We unified the existing definitions and concepts of dialogue
summarization into a taxonomy (CADS) of the field’s six main challenge blocks:
language, structure, comprehension, speaker, salience, and reliability-related
factuality. We demonstrated how these challenges appear in dialogue
summarization and discussed why they occur for established encoder-decoder-
based summarization systems. We then grouped and highlighted techniques
introduced since 2019 under the challenges they tackle. Despite advancements,
we observed that the field is still in its infancy and offers ample
opportunities for further research. We also listed datasets used in existing
studies to illustrate the extent of data scarcity in dialogue summarization.
Additionally, we evaluated common metrics and noted a strong reliance on the
ROUGE metric, coupled with a lack of human evaluation reports, which raises
doubts about the actual effectiveness of current techniques. In our
discussion, we evaluated how current NLP techniques address these challenges
and derived implications of using LLMs for the field. We thereby identify a
lack of exploring such new models and borrowing effective techniques from
related fields to overcome the limitations currently holding back progress. We
conclude that while challenges like language may become less relevant, others,
such as comprehension and factuality, still require more exploration. We
recommend that future work should further discuss and adapt this taxonomy as
new challenges, techniques, datasets, and evaluation measures emerge, and as
current challenges may be solved naturally through technological innovation.
The considered literature can be found in the accompanying repository
888https://anonymous.4open.science/r/LitRev-DialogueSum/  which will be
regularly updated.

Acknowledgments

This work was supported by the Lower Saxony Ministry of Science and Culture
and the VW Foundation. Frederic Kirstein was supported by the Mercedes-Benz AG
Research and Development.

## Glossary

LLM

    Large Language Model
NLP

    Natural Language Processing

## References

  * Abacha, Yim, Fan, and Lin Abacha, A. B., Yim, W.-w., Fan, Y., and Lin, T. (2023).  An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters.  In Conference of the European Chapter of the Association for Computational Linguistics.
  * Ainslie, Lei, de Jong, Ontañón, Brahma, Zemlyanskiy, Uthus, Guo, Lee-Thorp, Tay, Sung, and Sanghai Ainslie, J., Lei, T., de Jong, M., Ontañón, S., Brahma, S., Zemlyanskiy, Y., Uthus, D., Guo, M., Lee-Thorp, J., Tay, Y., Sung, Y.-H., and Sanghai, S. (2023).  CoLT5: Faster Long-Range Transformers with Conditional Computation..
  * Antony, Abhishek, Singh, Kodagali, Darapaneni, Rao, Paduri, and BG Antony, D., Abhishek, S., Singh, S., Kodagali, S., Darapaneni, N., Rao, M., Paduri, A. R., and BG, S. (2023).  A Survey of Advanced Methods for Efficient Text Summarization.  In 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0962–0968.
  * Asi, Wang, Eisenstadt, Geckt, Kuper, Mao, and Ronen Asi, A., Wang, S., Eisenstadt, R., Geckt, D., Kuper, Y., Mao, Y., and Ronen, R. (2022).  An End-to-End Dialogue Summarization System for Sales Calls..
  * Asthana, Hilleli, He, and Halfaker Asthana, S., Hilleli, S., He, P., and Halfaker, A. (2023).  Summaries, Highlights, and Action Items: Design, Implementation and Evaluation of an LLM-powered Meeting Recap System..  [object Object].
  * Bakker, Chadwick, Sheahan, Tessler, Campbell-Gillingham, Balaguer, McAleese, Glaese, Aslanides, Botvinick, and Summerfield Bakker, M. A., Chadwick, M. J., Sheahan, H. R., Tessler, M. H., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M. M., and Summerfield, C. (2022).  Fine-Tuning Language Models to Find Agreement among Humans with Diverse Preferences..
  * Balaguer, Benara, Cunha, Filho, Hendry, Holstein, Marsman, Mecklenburg, Malvar, Nunes, Padilha, Sharp, Silva, Sharma, Aski, and Chandra Balaguer, A., Benara, V., Cunha, R. L. d. F., Filho, R. d. M. E., Hendry, T., Holstein, D., Marsman, J., Mecklenburg, N., Malvar, S., Nunes, L. O., Padilha, R., Sharp, M., Silva, B., Sharma, S., Aski, V., and Chandra, R. (2024).  RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture..
  * Banerjee and Lavie Banerjee, S., and Lavie, A. (2005).  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72 Ann Arbor, Michigan. Association for Computational Linguistics.
  * Beckage, H Kumar, Sahay, and Manuvinakurike Beckage, N., H Kumar, S., Sahay, S., and Manuvinakurike, R. (2021).  Context or No Context? A Preliminary Exploration of Human-in-the-Loop Approach for Incremental Temporal Summarization in Meetings.  In Proceedings of the Third Workshop on New Frontiers in Summarization, pp. 96–106 Online and in Dominican Republic. Association for Computational Linguistics.
  * Beltagy, Peters, and Cohan Beltagy, I., Peters, M. E., and Cohan, A. (2020a).  Longformer: The Long-Document Transformer..
  * Beltagy, Peters, and Cohan Beltagy, I., Peters, M. E., and Cohan, A. (2020b).  Longformer: The Long-Document Transformer..
  * Bertsch, Neubig, and Gormley Bertsch, A., Neubig, G., and Gormley, M. R. (2022).  He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4823–4840 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bhatia, Biyani, and Mitra Bhatia, S., Biyani, P., and Mitra, P. (2014).  Summarizing Online Forum Discussions – Can Dialog Acts of Individual Messages Help?.  In Moschitti, A., Pang, B., and Daelemans, W. (Eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2127–2131 Doha, Qatar. Association for Computational Linguistics.
  * Bhatnagar, Khurana, and Kumar Bhatnagar, V., Khurana, A., and Kumar, V. (2023).  I Want This, Not That: Personalized Summarization of Scientific Scholarly Texts..
  * Biswas and Iakubovich Biswas, P. K., and Iakubovich, A. (2022).  Extractive Summarization of Call Transcripts.  IEEE Access, 10, 119826–119840.
  * Brereton, Kitchenham, Budgen, Turner, and Khalil Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M. (2007).  Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain.  Journal of Systems and Software, 80, 4, 571–583.
  * Budzianowski, Wen, Tseng, Casanueva, Ultes, Ramadan, and Gašić Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., and Gašić, M. (2018).  MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling.  In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016–5026 Brussels, Belgium. Association for Computational Linguistics.
  * Carbonell and Goldstein Carbonell, J., and Goldstein, J. (1998).  The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.  In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’98, pp. 335–336 New York, NY, USA. Association for Computing Machinery.
  * Carletta, Ashby, Bourban, Flynn, Guillemot, Hain, Kadlec, Karaiskos, Kraaij, Kronenthal, Lathoud, Lincoln, Lisowska, McCowan, Post, Reidsma, and Wellner Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., and Wellner, P. (2006).  The AMI Meeting Corpus: A Pre-announcement.  In Renals, S., and Bengio, S. (Eds.), Machine Learning for Multimodal Interaction, pp. 28–39 Berlin, Heidelberg. Springer.
  * Chauhan, Roy, Dey, and Goel Chauhan, V., Roy, P., Dey, L., and Goel, T. (2022).  TCS_WITM_2022 @ DialogSum : Topic Oriented Summarization Using Transformer Based Encoder Decoder Model.  In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges, pp. 104–109 Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
  * Chen, Dodda, and Yang Chen, J., Dodda, M., and Yang, D. (2023).  Human-in-the-Loop Abstractive Dialogue Summarization.  In Findings of the Association for Computational Linguistics: ACL 2023, pp. 9176–9190 Toronto, Canada. Association for Computational Linguistics.
  * Chen and Yang Chen, J., and Yang, D. (2020).  Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization..
  * Chen and Yang Chen, J., and Yang, D. (2021).  Simple Conversational Data Augmentation for Semi-supervised Abstractive Dialogue Summarization.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6605–6616 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chen, Chu, Wiseman, and Gimpel Chen, M., Chu, Z., Wiseman, S., and Gimpel, K. (2022).  SummScreen: A Dataset for Abstractive Screenplay Summarization.  Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8602–8615.  Association for Computational Linguistics, Dublin, Ireland.
  * Chen, Liu, Chen, and Zhang Chen, Y., Liu, Y., Chen, L., and Zhang, Y. (2021).  DialogSum: A Real-Life Scenario Dialogue Summarization Dataset..
  * Chen, Liu, Xu, Yang, Zhu, Zeng, and Zhang Chen, Y., Liu, Y., Xu, R., Yang, Z., Zhu, C., Zeng, M., and Zhang, Y. (2023).  UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12833–12855 Toronto, Canada. Association for Computational Linguistics.
  * Chiang and Lee Chiang, C.-H., and Lee, H.-y. (2023).  Can Large Language Models Be an Alternative to Human Evaluations?.  In Rogers, A., Boyd-Graber, J., and Okazaki, N. (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607–15631 Toronto, Canada. Association for Computational Linguistics.
  * Chintagunta, Katariya, Amatriain, and Kannan Chintagunta, B., Katariya, N., Amatriain, X., and Kannan, A. (2021).  Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization.  In Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations, pp. 66–76 Online. Association for Computational Linguistics.
  * Cohan, Dernoncourt, Kim, Bui, Kim, Chang, and Goharian Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. (2018).  A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents.  In Walker, M., Ji, H., and Stent, A. (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621 New Orleans, Louisiana. Association for Computational Linguistics.
  * Cohen Cohen, J. (1960).  A Coefficient of Agreement for Nominal Scales - Jacob Cohen, 1960.  https://journals.sagepub.com/doi/10.1177/001316446002000104.
  * Cui, Wu, Liu, Zhang, and Zhou Cui, L., Wu, Y., Liu, S., Zhang, Y., and Zhou, M. (2020).  MuTual: A Dataset for Multi-Turn Dialogue Reasoning.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1406–1416 Online. Association for Computational Linguistics.
  * Dang, Mecke, Lehmann, Goller, and Buschek Dang, H., Mecke, L., Lehmann, F., Goller, S., and Buschek, D. (2022).  How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models..
  * Dao, Fu, Ermon, Rudra, and Ré Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. (2022).  FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness..
  * Das, Kottur, Gupta, Singh, Yadav, Moura, Parikh, and Batra Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., and Batra, D. (2017).  Visual Dialog..
  * Deng, Cheng, Sun, Zhang, and Huang Deng, J., Cheng, J., Sun, H., Zhang, Z., and Huang, M. (2023).  Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements..
  * Deutsch, Bedrax-Weiss, and Roth Deutsch, D., Bedrax-Weiss, T., and Roth, D. (2021).  Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary..
  * Devlin, Chang, Lee, and Toutanova Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019).  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong, Li, Gong, Chen, Li, Shen, and Yang Dong, C., Li, Y., Gong, H., Chen, M., Li, J., Shen, Y., and Yang, M. (2022).  A Survey of Natural Language Generation.  ACM Computing Surveys, 55, 8, 173:1–173:38.
  * Dong, Yang, Wang, Wei, Liu, Wang, Gao, Zhou, and Hon Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. (2019).  Unified Language Model Pre-training for Natural Language Understanding and Generation..
  * Dou, Forbes, Koncel-Kedziorski, Smith, and Choi Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., and Choi, Y. (2022).  Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
  * Duan, Zhang, Yuan, Zhou, Liu, Wang, Wang, Zhang, Sun, and Wu Duan, X., Zhang, Y., Yuan, L., Zhou, X., Liu, X., Wang, T., Wang, R., Zhang, Q., Sun, C., and Wu, F. (2019).  Legal Summarization for Multi-role Debate Dialogue via Controversy Focus Mining and Multi-task Learning.  Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1361–1370.  ACM, Beijing China.
  * Durmus, He, and Diab Durmus, E., He, H., and Diab, M. (2020).  FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5055–5070 Online. Association for Computational Linguistics.
  * Elder, Gehrmann, O’Connor, and Liu Elder, H., Gehrmann, S., O’Connor, A., and Liu, Q. (2018).  E2E NLG Challenge Submission: Towards Controllable Generation of Diverse Natural Language.  In Proceedings of the 11th International Conference on Natural Language Generation. Association for Computational Linguistics.
  * Fabbri, Rahman, Rizvi, Wang, Li, Mehdad, and Radev Fabbri, A., Rahman, F., Rizvi, I., Wang, B., Li, H., Mehdad, Y., and Radev, D. (2021a).  ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6866–6880 Online. Association for Computational Linguistics.
  * Fabbri, Kryściński, McCann, Xiong, Socher, and Radev Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., and Radev, D. (2021b).  SummEval: Re-evaluating Summarization Evaluation.  Transactions of the Association for Computational Linguistics, 9, 391–409.
  * Fagan Fagan, J. C. (2017).  An Evidence-Based Review of Academic Web Search Engines, 2014-2016: Implications for Librarians’ Practice and Research Agenda.  Information Technology and Libraries, 36, 2, 7–47.  American Library Association.
  * Fang, Pan, Zhang, Song, Xu, and Yu Fang, T., Pan, H., Zhang, H., Song, Y., Xu, K., and Yu, D.  Do Boat and Ocean Suggest Beach? Dialogue Summarization with External Knowledge..
  * Fang, Zhang, Chen, Ding, Long, Lan, and Zhou Fang, Y., Zhang, H., Chen, H., Ding, Z., Long, B., Lan, Y., and Zhou, Y. (2022).  From Spoken Dialogue to Formal Summary: An Utterance Rewriting for Dialogue Summarization.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3859–3869 Seattle, United States. Association for Computational Linguistics.
  * Fazly, Cook, and Stevenson Fazly, A., Cook, P., and Stevenson, S. (2009).  Unsupervised Type and Token Identification of Idiomatic Expressions.  Computational Linguistics, 35, 1, 61–103.
  * Feigenblat, Gunasekara, Sznajder, Joshi, Konopnicki, and Aharonov Feigenblat, G., Gunasekara, C., Sznajder, B., Joshi, S., Konopnicki, D., and Aharonov, R. (2021).  TWEETSUMM - A Dialog Summarization Dataset for Customer Service.  Findings of the Association for Computational Linguistics: EMNLP 2021, 245–260.  Association for Computational Linguistics, Punta Cana, Dominican Republic.
  * Feng, Feng, and Qin Feng, X., Feng, X., and Qin, B. (2021).  Incorporating Commonsense Knowledge into Abstractive Dialogue Summarization via Heterogeneous Graph Networks.  Chinese Computational Linguistics, 12869, 127–142.  Springer International Publishing, Cham.
  * Feng, Feng, and Qin Feng, X., Feng, X., and Qin, B. (2022).  A Survey on Dialogue Summarization: Recent Advances and New Frontiers..
  * Feng, Feng, Qin, Qin, and Liu Feng, X., Feng, X., Qin, L., Qin, B., and Liu, T. (2021).  Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1479–1491 Online. Association for Computational Linguistics.
  * Finn and Louviere Finn, A., and Louviere, J. J. (1992).  Determining the Appropriate Response to Evidence of Public Concern: The Case of Food Safety.  Journal of Public Policy & Marketing, 11, 2, 12–25.  American Marketing Association.
  * Fleiss Fleiss, J. L. (1971).  Measuring Nominal Scale Agreement among Many Raters.  Psychological Bulletin, 76, 5, 378–382.  American Psychological Association, US.
  * Foltýnek, Meuschke, and Gipp Foltýnek, T., Meuschke, N., and Gipp, B. (2019).  Academic Plagiarism Detection: A Systematic Literature Review.  ACM Computing Surveys, 52, 6, 112:1–112:42.
  * Fu, Ng, Jiang, and Liu Fu, J., Ng, S.-K., Jiang, Z., and Liu, P. (2023).  GPTScore: Evaluate as You Desire..
  * Fu, Zhang, Wang, Liu, Sun, and Yang Fu, X., Zhang, Y., Wang, T., Liu, X., Sun, C., and Yang, Z. (2021).  RepSum: Unsupervised Dialogue Summarization Based on Replacement Strategy.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6042–6051 Online. Association for Computational Linguistics.
  * Gan, Zhang, Kuang, Yuan, Li, Sun, Liu, and Wu Gan, L., Zhang, Y., Kuang, K., Yuan, L., Li, S., Sun, C., Liu, X., and Wu, F. (2021).  Dialogue Inspectional Summarization with Factual Inconsistency Awareness..
  * Ganesh and Dingliwal Ganesh, P., and Dingliwal, S. (2019).  Restructuring Conversations Using Discourse Relations for Zero-shot Abstractive Dialogue Summarization..  arXiv: Computation and Language.
  * Gao, Wan, Su, Wang, and Huai Gao, M., Wan, X., Su, J., Wang, Z., and Huai, B. (2023).  Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework | Semantic Scholar.  https://www.semanticscholar.org/reader/8532a206424a33d9803752474d74b602ff8ea211.
  * Gao and Wan Gao, M., and Wan, X. (2022).  DialSummEval: Revisiting Summarization Evaluation for Dialogues.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5693–5709 Seattle, United States. Association for Computational Linguistics.
  * Gao, Cheng, Li, Chen, Li, Zhao, and Yan Gao, S., Cheng, X., Li, M., Chen, X., Li, J., Zhao, D., and Yan, R. (2023).  Dialogue Summarization with Static-Dynamic Structure Fusion Graph.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13858–13873 Toronto, Canada. Association for Computational Linguistics.
  * Gatt and Krahmer Gatt, A., and Krahmer, E. (2018).  Survey of the State of the Art in Natural Language Generation: Core Tasks, Applications and Evaluation.  Journal of Artificial Intelligence Research, 61, 65–170.  AI Access Foundation.
  * Geng, Zhong, Yin, Qiu, and Huang Geng, Z., Zhong, M., Yin, Z., Qiu, X., and Huang, X. (2022).  Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning.  In Proceedings of the 29th International Conference on Computational Linguistics, pp. 6540–6546 Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Gliwa, Mochol, Biesek, and Wawer Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. (2019).  SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization.  Proceedings of the 2nd Workshop on New Frontiers in Summarization, 70–79.  Association for Computational Linguistics, Hong Kong, China.
  * Gloeckle, Idrissi, Rozière, Lopez-Paz, and Synnaeve Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., and Synnaeve, G. (2024).  Better & Faster Large Language Models via Multi-token Prediction..
  * Gu, Tao, and Ling Gu, J.-C., Tao, C., and Ling, Z.-H. (2022).  Who Says What to Whom: A Survey of Multi-Party Conversations.  In Thirty-First International Joint Conference on Artificial Intelligence, Vol. 6, pp. 5486–5493.
  * Hannousse Hannousse, A. (2021).  Searching Relevant Papers for Software Engineering Secondary Studies: Semantic Scholar Coverage and Identification Role.  IET Software, 15, 1, 126–146.
  * Hospedales, Antoniou, Micaelli, and Storkey Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. (2020).  Meta-Learning in Neural Networks: A Survey..
  * Hossain, Hamilton, Palmer, and Blanco Hossain, M. M., Hamilton, K., Palmer, A., and Blanco, E. (2020).  Predicting the Focus of Negation: Model and Error Analysis.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8389–8401 Online. Association for Computational Linguistics.
  * Hu, Ganter, Deilamsalehy, Dernoncourt, Foroosh, and Liu Hu, Y., Ganter, T., Deilamsalehy, H., Dernoncourt, F., Foroosh, H., and Liu, F. (2023).  MeetingBank: A Benchmark Dataset for Meeting Summarization.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16409–16423 Toronto, Canada. Association for Computational Linguistics.
  * Hua, Deng, and McKeown Hua, Y., Deng, Z., and McKeown, K. (2023).  Improving Long Dialogue Summarization with Semantic Graph Representation.  In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13851–13883 Toronto, Canada. Association for Computational Linguistics.
  * Hua, Deng, and Xu Hua, Y., Deng, Z., and Xu, Z. (2022).  AMRTVSumm: AMR-augmented Hierarchical Network for TV Transcript Summarization.  In Mckeown, K. (Ed.), Proceedings of The Workshop on Automatic Summarization for Creative Writing, pp. 36–43 Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Huang, Singh, Ma, Xiao, Nan, Dingwall, Wang, and McKeown Huang, K.-H., Singh, S., Ma, X., Xiao, W., Nan, F., Dingwall, N., Wang, W. Y., and McKeown, K. (2023).  SWING: Balancing Coverage and Faithfulness for Dialogue Summarization..
  * Ibrahim and Shafiq Ibrahim, R., and Shafiq, M. O. (2023).  Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions.  ACM Computing Surveys, 55, 10, 206:1–206:37.
  * Ibrahim Altmami and El Bachir Menai Ibrahim Altmami, N., and El Bachir Menai, M. (2022).  Automatic Summarization of Scientific Articles: A Survey.  Journal of King Saud University - Computer and Information Sciences, 34, 4, 1011–1028.
  * Ioannides, Jadhav, Sharma, Navali, and Black Ioannides, G., Jadhav, A., Sharma, A., Navali, S., and Black, A. W. (2024).  Compressed Models for Co-reference Resolution: Enhancing Efficiency with Debiased Word Embeddings..
  * Jain, Keshava, Sathyendra, Fernandes, Liu, Neubig, and Zhou Jain, S., Keshava, V., Sathyendra, S. M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C. (2023).  Multi-Dimensional Evaluation of Text Summarization with In-Context Learning.  In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8487–8495.
  * Janin, Baron, Edwards, Ellis, Gelbart, Morgan, Peskin, Pfau, Shriberg, Stolcke, and Wooters Janin, A., Baron, D., Edwards, J., Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E., Stolcke, A., and Wooters, C. (2003).  The ICSI Meeting Corpus.  In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03)., Vol. 1, pp. I–I.
  * Jelinek, Mercer, Bahl, and Baker Jelinek, F., Mercer, R. L., Bahl, L. R., and Baker, J. K. (1977).  Perplexity—a Measure of the Difficulty of Speech Recognition Tasks.  The Journal of the Acoustical Society of America, 62, S1, S63-S63.
  * Ji, Lee, Frieske, Yu, Su, Xu, Ishii, Bang, Chen, Chan, Dai, Madotto, and Fung Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Chan, H. S., Dai, W., Madotto, A., and Fung, P. (2023).  Survey of Hallucination in Natural Language Generation.  ACM Computing Surveys, 55, 12, 1–38.
  * Jia, Liu, Tang, and Zhu Jia, Q., Liu, Y., Tang, H., and Zhu, K. (2022a).  Post-Training Dialogue Summarization Using Pseudo-Paraphrasing.  In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1660–1669 Seattle, United States. Association for Computational Linguistics.
  * Jia, Ren, Liu, and Zhu Jia, Q., Ren, S., Liu, Y., and Zhu, K. Q. (2022b).  Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions..
  * Jones, Ravid, and Rafaeli Jones, Q., Ravid, G., and Rafaeli, S. (2004).  Information Overload and the Message Dynamics of Online Interaction Spaces: A Theoretical Model and Empirical Exploration.  Information Systems Research, 15, 2, 194–210.  Institute for Operations Research & the Management Sciences (INFORMS), US.
  * Joshi, Chen, Liu, Weld, Zettlemoyer, and Levy Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. (2020).  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  Transactions of the Association for Computational Linguistics, 8, 64–77.  MIT Press, Cambridge, MA.
  * Jung, Seo, Jung, Chung, Ryu, and Chang Jung, J., Seo, H., Jung, S., Chung, R., Ryu, H., and Chang, D.-S. (2023).  Interactive User Interface for Dialogue Summarization.  In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 934–957 Sydney NSW Australia. ACM.
  * Khalifa, Ballesteros, and McKeown Khalifa, M., Ballesteros, M., and McKeown, K. (2021a).  A Bag of Tricks for Dialogue Summarization.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8014–8022 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Khalifa, Ballesteros, and McKeown Khalifa, M., Ballesteros, M., and McKeown, K. (2021b).  A Bag of Tricks for Dialogue Summarization..
  * Khurana, Koli, Khatter, and Singh Khurana, D., Koli, A., Khatter, K., and Singh, S. (2023).  Natural Language Processing: State of the Art, Current Trends and Challenges.  Multimedia Tools and Applications, 82, 3, 3713–3744.
  * King, Shen, Subramani, Weld, Beltagy, and Downey King, D., Shen, Z., Subramani, N., Weld, D. S., Beltagy, I., and Downey, D. (2023).  Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search..
  * Kiritchenko and Mohammad Kiritchenko, S., and Mohammad, S. (2017).  Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation.  In Barzilay, R., and Kan, M.-Y. (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 465–470 Vancouver, Canada. Association for Computational Linguistics.
  * Kirstein, Wahle, Ruas, and Gipp Kirstein, F., Wahle, J. P., Ruas, T., and Gipp, B. (2024).  What’s under the Hood: Investigating Automatic Metrics on Meeting Summarization..
  * Kitchenham Kitchenham, B.  Procedures for Performing Systematic Reviews..
  * Koay, Roustai, Dai, Burns, Kerrigan, and Liu Koay, J. J., Roustai, A., Dai, X., Burns, D., Kerrigan, A., and Liu, F. (2020).  How Domain Terminology Affects Meeting Summarization Performance.  In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5689–5695 Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kocmi and Federmann Kocmi, T., and Federmann, C. (2023).  Large Language Models Are State-of-the-Art Evaluators of Translation Quality..
  * Krippendorff Krippendorff, K. (1970).  Bivariate Agreement Coefficients for Reliability of Data.  Sociological Methodology, 2, 139–150.  [American Sociological Association, Wiley, Sage Publications, Inc.].
  * Kryscinski, Keskar, McCann, Xiong, and Socher Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., and Socher, R. (2019).  Neural Text Summarization: A Critical Evaluation.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 540–551 Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski, McCann, Xiong, and Socher Kryscinski, W., McCann, B., Xiong, C., and Socher, R. (2020).  Evaluating the Factual Consistency of Abstractive Text Summarization.  In Webber, B., Cohn, T., He, Y., and Liu, Y. (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332–9346 Online. Association for Computational Linguistics.
  * Kumar and Kabiri Kumar, L. P., and Kabiri, A. (2022).  Meeting Summarization: A Survey of the State of the Art..
  * Kusner, Sun, Kolkin, and Weinberger Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K. (2015).  From Word Embeddings To Document Distances.  In Proceedings of the 32nd International Conference on Machine Learning, pp. 957–966. PMLR.
  * Laskar, Fu, Chen, and Bhushan Tn Laskar, M. T. R., Fu, X.-Y., Chen, C., and Bhushan Tn, S. (2023).  Building Real-World Meeting Summarization Systems Using Large Language Models: A Practical Perspective.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 343–352 Singapore. Association for Computational Linguistics.
  * Lee, Lim, Whang, Lee, Cho, Park, and Lim Lee, D., Lim, J., Whang, T., Lee, C., Cho, S., Park, M., and Lim, H. (2021a).  Capturing Speaker Incorrectness: Speaker-Focused Post-Correction for Abstractive Dialogue Summarization.  Proceedings of the Third Workshop on New Frontiers in Summarization, 65–73.  Association for Computational Linguistics, Online and in Dominican Republic.
  * Lee, Yun, Choi, Joe, and Gwon Lee, H., Yun, J., Choi, H., Joe, S., and Gwon, Y. L. (2021b).  Enhancing Semantic Understanding with Self-Supervised Methods for Abstractive Dialogue Summarization.  In Interspeech 2021, pp. 796–800. ISCA.
  * Lee, Kayaalp, Henry, and Uzuner Lee, K., Kayaalp, M., Henry, S., and Uzuner, Ö. (2021c).  A Context-Enhanced De-identification System.  ACM Transactions on Computing for Healthcare, 3, 1, 6:1–6:14.
  * Lee, Yang, Park, Sedoc, and Lim Lee, S., Yang, K., Park, C., Sedoc, J., and Lim, H. (2021d).  Who Speaks Like a Style of Vitamin: Towards Syntax-Aware Dialogue Summarization Using Multi-Task Learning.  IEEE Access, 9, 168889–168898.
  * Lei, Yan, Zeng, He, Zhang, and XuS Lei, Y., Yan, Y., Zeng, Z., He, K., Zhang, X., and XuS, W. (2021a).  Hierarchical Speaker-Aware Sequence-to-Sequence Model for Dialogue Summarization.  ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7823–7827.  IEEE, Toronto, ON, Canada.
  * Lei, Zheng, Yan, He, and Xu Lei, Y., Zheng, F., Yan, Y., He, K., and Xu, W. (2021b).  A Finer-grain Universal Dialogue Semantic Structures Based Model For Abstractive Dialogue Summarization.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1354–1364 Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov, and Zettlemoyer Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020a).  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880 Online. Association for Computational Linguistics.
  * Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov, and Zettlemoyer Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020b).  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880 Online. Association for Computational Linguistics.
  * Li, Wang, Guo, Song, Tan, Hassan, Menezes, Xiao, Bian, and Zhu Li, B., Wang, R., Guo, J., Song, K., Tan, X., Hassan, H., Menezes, A., Xiao, T., Bian, J., and Zhu, J. (2023).  Deliberate Then Generate: Enhanced Prompting Framework for Text Generation..
  * Li, Wang, Lin, de Melo, and He Li, C., Wang, L., Lin, X., de Melo, G., and He, L. (2022).  Curriculum Prompt Learning with Self-Training for Abstractive Dialogue Summarization.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1096–1106 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Li, Huber, Xiao, Amblard, Braud, and Carenini Li, C., Huber, P., Xiao, W., Amblard, M., Braud, C., and Carenini, G. (2023).  Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues.  In Findings of the Association for Computational Linguistics: EACL 2023, pp. 2562–2579 Dubrovnik, Croatia. Association for Computational Linguistics.
  * Li, Xu, Yuan, Wang, Wu, He, and Zhou Li, H., Xu, S., Yuan, P., Wang, Y., Wu, Y., He, X., and Zhou, B. (2021).  Learn to Copy from the Copying History: Correlational Copy Network for Abstractive Summarization.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4091–4101 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li Li, H. (2022).  URAMDS: Utterances Relation Aware Model for Dialogue Summarization: A Combined Model for Dialogue Summarization.  In 2022 2nd International Conference on Bioinformatics and Intelligent Computing, pp. 397–401 Harbin China. ACM.
  * Li, Xia, Cheng, Zhao, and Yan Li, J., Xia, Y., Cheng, X., Zhao, D., and Yan, R. (2023).  Learning Disentangled Representation via Domain Adaptation for Dialogue Summarization.  In Proceedings of the ACM Web Conference 2023, pp. 1693–1702 Austin TX USA. ACM.
  * Li and Liang Li, X. L., and Liang, P. (2021).  Prefix-Tuning: Optimizing Continuous Prompts for Generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics.
  * Li, Su, Shen, Li, Cao, and Niu Li, Y., Su, H., Shen, X., Li, W., Cao, Z., and Niu, S. (2017).  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset.  In Kondrak, G., and Watanabe, T. (Eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 986–995 Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Liang, Wu, Cui, Bai, Bian, and Li Liang, X., Wu, S., Cui, C., Bai, J., Bian, C., and Li, Z. (2023).  Enhancing Dialogue Summarization with Topic-Aware Global- and Local- Level Centrality.  In Vlachos, A., and Augenstein, I. (Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 27–38 Dubrovnik, Croatia. Association for Computational Linguistics.
  * Likert Likert, R. (1932).  A Technique for the Measurement of Attitudes.  Archives of Psychology, 22 140, 55–55.
  * Lin Lin, C.-Y. (2004).  ROUGE: A Package for Automatic Evaluation of Summaries.  In Text Summarization Branches Out, pp. 74–81 Barcelona, Spain. Association for Computational Linguistics.
  * Lin, Ma, Zhu, Xiang, Zhou, Zhang, and Zong Lin, H., Ma, L., Zhu, J., Xiang, L., Zhou, Y., Zhang, J., and Zong, C. (2021).  CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization.  In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4436–4451 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lin, Zhu, Xiang, Zhou, Zhang, and Zong Lin, H., Zhu, J., Xiang, L., Zhou, Y., Zhang, J., and Zong, C. (2022).  Other Roles Matter! Enhancing Role-Oriented Dialogue Summarization via Role Interactions..  arXiv.
  * Lin, Maire, Belongie, Bourdev, Girshick, Hays, Perona, Ramanan, Zitnick, and Dollár Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. (2015).  Microsoft COCO: Common Objects in Context..
  * Liu, Sap, Lu, Swayamdipta, Bhagavatula, Smith, and Choi Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. (2021).  DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics.
  * Liu, Wang, Xu, Li, and Ye Liu, C., Wang, P., Xu, J., Li, Z., and Ye, J. (2019).  Automatic Dialogue Summary Generation for Customer Service.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1957–1965 Anchorage AK USA. ACM.
  * Liu and Singh Liu, H., and Singh, P. (2004).  ConceptNet — A Practical Commonsense Reasoning Tool-Kit.  BT Technology Journal, 22, 4, 211–226.
  * Liu, Zaharia, and Abbeel Liu, H., Zaharia, M., and Abbeel, P. (2023).  Ring Attention with Blockwise Transformers for Near-Infinite Context..
  * Liu, Zou, Xi, Li, Ma, and Ding Liu, J., Zou, Y., Xi, Y., Li, S., Ma, M., and Ding, Z. (2022a).  Summarizing Dialogues with Negative Cues.  In Calzolari, N., Huang, C.-R., Kim, H., Pustejovsky, J., Wanner, L., Choi, K.-S., Ryu, P.-M., Chen, H.-H., Donatelli, L., Ji, H., Kurohashi, S., Paggio, P., Xue, N., Kim, S., Hahm, Y., He, Z., Lee, T. K., Santus, E., Bond, F., and Na, S.-H. (Eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 6050–6056 Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Liu, Zou, Xi, Li, Ma, Ding, and Long Liu, J., Zou, Y., Xi, Y., Li, S., Ma, M., Ding, Z., and Long, B. (2022b).  Negative Guided Abstractive Dialogue Summarization.  In Interspeech 2022, pp. 3253–3257. ISCA.
  * Liu, Zou, Zhang, Chen, Ding, Yuan, and Wang Liu, J., Zou, Y., Zhang, H., Chen, H., Ding, Z., Yuan, C., and Wang, X. (2021).  Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization..
  * Liu, Mao, Luu, and Cambria Liu, R., Mao, R., Luu, A. T., and Cambria, E. (2023).  A Brief Survey on Recent Advances in Coreference Resolution.  Artificial Intelligence Review, 56, 12, 14439–14481.
  * Liu, Maynez, Simões, and Narayan Liu, Y., Maynez, J., Simões, G., and Narayan, S. (2022).  Data Augmentation for Low-Resource Dialogue Summarization.  In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 703–710 Seattle, United States. Association for Computational Linguistics.
  * Liu, Yu, and Lin Liu, Y., Yu, S., and Lin, T. (2023).  Hessian Regularization of Deep Neural Networks: A Novel Approach Based on Stochastic Estimators of Hessian Trace.  Neurocomputing, 536, 13–20.
  * Liu and Chen Liu, Z., and Chen, N. (2022).  Entity-Based De-noising Modeling for Controllable Dialogue Summarization.  In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 407–418 Edinburgh, UK. Association for Computational Linguistics.
  * Liu and Chen Liu, Z., and Chen, N. F. (2021).  Controllable Neural Dialogue Summarization with Personal Named Entity Planning..
  * Liu, Shi, and Chen Liu, Z., Shi, K., and Chen, N. F. (2021).  Coreference-Aware Dialogue Summarization..
  * Lu, Welleck, West, Jiang, Kasai, Khashabi, Bras, Qin, Yu, Zellers, Smith, and Choi Lu, X., Welleck, S., West, P., Jiang, L., Kasai, J., Khashabi, D., Bras, R. L., Qin, L., Yu, Y., Zellers, R., Smith, N., and Choi, Y. (2022).  NeuroLogic Aesque Decoding: Constrained Text Generation with Lookahead Heuristics.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
  * Lu, Bo, and He Lu, Y., Bo, Y., and He, W. (2021).  Confidence Adaptive Regularization for Deep Learning with Noisy Labels..
  * Lyu, Du, Xu, Duan, Wu, Lynn, Aji, Wong, and Wang Lyu, C., Du, Z., Xu, J., Duan, Y., Wu, M., Lynn, T., Aji, A. F., Wong, D. F., and Wang, L. (2024a).  A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models..
  * Lyu, Peng, Li, Balian, Bian, and Wu Lyu, M., Peng, C. A. I., Li, X., Balian, P., Bian, J., and Wu, Y. (2024b).  Automatic Summarization of Doctor-Patient Encounter Dialogues Using Large Language Model through Prompt Tuning..
  * Ma, Zhang, Guo, Wang, and Sheng Ma, C., Zhang, W. E., Guo, M., Wang, H., and Sheng, Q. Z. (2023).  Multi-Document Summarization via Deep Learning Techniques: A Survey.  ACM Computing Surveys, 55, 5, 1–37.
  * Ma, Zhou, Gui, Tan, Li, Zhang, and Huang Ma, R., Zhou, X., Gui, T., Tan, Y., Li, L., Zhang, Q., and Huang, X. (2022).  Template-Free Prompt Tuning for Few-shot NER..
  * Majumder, Li, Ni, and McAuley Majumder, B. P., Li, S., Ni, J., and McAuley, J. (2020).  Interview: A Large-Scale Open-Source Corpus of Media Dialog..
  * Mane, Khadtare, Hingmire, Pawar, and Vasal Mane, Khadtare, Hingmire, Pawar, and Vasal.  Exploring Advances in Meeting Minutes Generation and Face Attendance Systems: A Comprehensive Literature Survey..
  * Manuvinakurike, Sahay, Chen, and Nachman Manuvinakurike, R., Sahay, S., Chen, W., and Nachman, L. (2021).  Incremental Temporal Summarization in Multi-Party Meetings.  In SIGDIAL Conferences.
  * McIntosh, Liu, Susnjak, Watters, Ng, and Halgamuge McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., and Halgamuge, M. N. (2023).  A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination.  https://www.computer.org/csdl/journal/ai/5555/01/10319443/1S5SVt0zLnG.
  * Meskill Meskill, C. (1993).  ESL and Multimedia: A Study of the Dynamics of Paired Student Discourse.  System, 21, 3, 323–341.
  * Misra, Anand, Fox Tree, and Walker Misra, A., Anand, P., Fox Tree, J. E., and Walker, M. (2015).  Using Summarization to Discover Argument Facets in Online Idealogical Dialog.  In Mihalcea, R., Chai, J., and Sarkar, A. (Eds.), Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 430–440 Denver, Colorado. Association for Computational Linguistics.
  * Moramarco, Juric, Savkov, Flann, Lehl, Boda, Grafen, Zhelezniak, Gohil, Korfiatis, and Hammerla Moramarco, F., Juric, D., Savkov, A., Flann, J., Lehl, M., Boda, K., Grafen, T., Zhelezniak, V., Gohil, S., Korfiatis, A. P., and Hammerla, N. (2021).  Towards More Patient Friendly Clinical Notes through Language Models and Ontologies.  AMIA … Annual Symposium proceedings. AMIA Symposium, 2021, 881–890.
  * Mullick, Bhowmick, R, Kokku, Dey, Goyal, and Ganguly Mullick, A., Bhowmick, A. K., R, R., Kokku, R., Dey, P., Goyal, P., and Ganguly, N. (2024).  Long Dialog Summarization: An Analysis..  [object Object].
  * Nair, Schumacher, and Kannan Nair, V., Schumacher, E., and Kannan, A. (2023a).  Generating Medically-Accurate Summaries of Patient-Provider Dialogue: A Multi-Stage Approach Using Large Language Models..  arXiv.
  * Nair, Schumacher, and Kannan Nair, V., Schumacher, E., and Kannan, A. (2023b).  Generating Medically-Accurate Summaries of Patient-Provider Dialogue: A Multi-Stage Approach Using Large Language Models..  [object Object].
  * Naraki, Sakai, and Hayashi Naraki, Y., Sakai, T., and Hayashi, Y. (2022).  Evaluating the Effects of Embedding with Speaker Identity Information in Dialogue Summarization.  In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 298–304 Marseille, France. European Language Resources Association.
  * Nedoluzhko, Singh, Hledíková, Ghosal, and Bojar Nedoluzhko, A., Singh, M., Hledíková, M., Ghosal, T., and Bojar, O. (2022).  ELITR Minuting Corpus: A Novel Dataset for Automatic Minuting from Multi-Party Meetings in English and Czech.  In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 3174–3182 Marseille, France. European Language Resources Association.
  * Neto, Freitas, and Kaestner Neto, J. L., Freitas, A. A., and Kaestner, C. A. A. (2002).  Automatic Text Summarization Using a Machine Learning Approach.  In Bittencourt, G., and Ramalho, G. L. (Eds.), Advances in Artificial Intelligence, pp. 205–215 Berlin, Heidelberg. Springer.
  * Ng and Abrecht Ng, J.-P., and Abrecht, V. (2015).  Better Summarization Evaluation with Word Embeddings for ROUGE.  In Màrquez, L., Callison-Burch, C., and Su, J. (Eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1925–1930 Lisbon, Portugal. Association for Computational Linguistics.
  * Nguyen, Jatowt, Coustaty, and Doucet Nguyen, T. T. H., Jatowt, A., Coustaty, M., and Doucet, A. (2021).  Survey of Post-OCR Processing Approaches.  ACM Computing Surveys, 54, 6, 124:1–124:37.
  * Page, McKenzie, Bossuyt, Boutron, Hoffmann, Mulrow, Shamseer, Tetzlaff, Akl, Brennan, Chou, Glanville, Grimshaw, Hróbjartsson, Lalu, Li, Loder, Mayo-Wilson, McDonald, McGuinness, Stewart, Thomas, Tricco, Welch, Whiting, and Moher Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., and Moher, D. (2021).  The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews.  Systematic Reviews, 10, 1, 89.
  * Pagnoni, Fabbri, Kryściński, and Wu Pagnoni, A., Fabbri, A. R., Kryściński, W., and Wu, C.-S. (2022).  Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization..  arXiv.
  * Pakhale Pakhale, K. (2023).  Comprehensive Overview of Named Entity Recognition: Models, Domain-Specific Applications and Challenges..
  * Papineni, Roukos, Ward, and Zhu Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002).  Bleu: A Method for Automatic Evaluation of Machine Translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Park and Storey Park, E. H., and Storey, V. C. (2023).  Emotion Ontology Studies: A Framework for Expressing Feelings Digitally and Its Application to Sentiment Analysis.  ACM Computing Surveys, 55, 9, 181:1–181:38.
  * Park, Shin, and Lee Park, S., Shin, D., and Lee, J. (2022).  Leveraging Non-dialogue Summaries for Dialogue Summarization.  In Proceedings of the First Workshop On Transcript Understanding, pp. 1–7 Gyeongju, South Korea. International Conference on Computational Linguistics.
  * Peysakhovich and Lerer Peysakhovich, A., and Lerer, A. (2023).  Attention Sorting Combats Recency Bias In Long Context Language Models..
  * Popović Popović, M. (2015).  chrF: Character n-Gram F-score for Automatic MT Evaluation.  In Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Hokamp, C., Huck, M., Logacheva, V., and Pecina, P. (Eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392–395 Lisbon, Portugal. Association for Computational Linguistics.
  * Popović Popović, M. (2017).  chrF++: Words Helping Character n-Grams.  In Proceedings of the Second Conference on Machine Translation, pp. 612–618 Copenhagen, Denmark. Association for Computational Linguistics.
  * Prodan and Pelican Prodan, G., and Pelican, E. (2022).  Prompt Scoring System for Dialogue Summarization Using GPT-3..
  * Qader, Jneid, Portet, and Labbé Qader, R., Jneid, K., Portet, F., and Labbé, C. (2018).  Generation of Company Descriptions Using Concept-to-Text and Text-to-Text Deep Models: Dataset Collection and Systems Evaluation.  In Proceedings of the 11th International Conference on Natural Language Generation. Association for Computational Linguistics.
  * Qi, Liu, Fu, and Liu Qi, M., Liu, H., Fu, Y., and Liu, T. (2021).  Improving Abstractive Dialogue Summarization with Hierarchical Pretraining and Topic Segment.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1121–1130 Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Qin and Joty Qin, C., and Joty, S. (2022).  LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5..
  * Quan, Xiong, Webber, and Hu Quan, J., Xiong, D., Webber, B., and Hu, C. (2019).  GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue.  In Inui, K., Jiang, J., Ng, V., and Wan, X. (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4547–4557 Hong Kong, China. Association for Computational Linguistics.
  * R, Ramesh, M, R, Ramesh, and M R, V., Ramesh, D., M, H., R, V., Ramesh, D., and M, H. (2023).  Automatic Text Summarization–A Systematic Literature Review.  World Journal of Advanced Engineering Technology and Sciences, 8, 2, 126–129.  World Journal of Advanced Engineering Technology and Sciences.
  * Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li, and Liu Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020).  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer..
  * Rai and Chakraverty Rai, S., and Chakraverty, S. (2020).  A Survey on Computational Metaphor Processing.  ACM Computing Surveys, 53, 2, 24:1–24:37.
  * Rameshkumar and Bailey Rameshkumar, R., and Bailey, P. (2020).  Storytelling with Dialogue: A Critical Role Dungeons and Dragons Dataset.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5121–5134 Online. Association for Computational Linguistics.
  * Ravaut, Joty, and Chen Ravaut, M., Joty, S., and Chen, N. (2022).  SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization.  In Muresan, S., Nakov, P., and Villavicencio, A. (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4504–4524 Dublin, Ireland. Association for Computational Linguistics.