# Text Generation: A Systematic Literature Review of Tasks, Evaluation, and
Challenges

Jonas Becker<EMAIL_ADDRESS>University of
GöttingenGöttingenGermany 0009-0006-6438-1211 ,  Jan Philip Wahle  wahle@uni-
goettingen.de University of GöttingenGöttingenGermany 0000-0002-2116-9767 ,
Bela Gipp  University of GöttingenGöttingenGermany<EMAIL_ADDRESS>0000-0001-6522-3019 and  Terry Ruas  University of GöttingenGöttingenGermany
<EMAIL_ADDRESS>0000-0002-9440-780X

(2024; 20 February 2007; 12 March XXXX; 5 June XXXX)

###### Abstract.

Text generation has become more accessible than ever, and the increasing
interest in these systems, especially those using large language models, has
spurred an increasing number of related publications. We provide a systematic
literature review comprising 244 selected papers between 2017 and 2024. This
review categorizes works in text generation into five main tasks: open-ended
text generation, summarization, translation, paraphrasing, and question
answering. For each task, we review their relevant characteristics, sub-tasks,
and specific challenges (e.g., missing datasets for multi-document
summarization, coherence in story generation, and complex reasoning for
question answering). Additionally, we assess current approaches for evaluating
text generation systems and ascertain problems with current metrics. Our
investigation shows nine prominent challenges common to all tasks and sub-
tasks in recent text generation publications: bias, reasoning, hallucinations,
misuse, privacy, interpretability, transparency, datasets, and computing. We
provide a detailed analysis of these challenges, their potential solutions,
and which gaps still require further engagement from the community. This
systematic literature review targets two main audiences: early career
researchers in natural language processing looking for an overview of the
field and promising research directions, as well as experienced researchers
seeking a detailed view of tasks, evaluation methodologies, open challenges,
and recent mitigation strategies.

text generation, machine-generated text, language model, natural language
generation, natural language processing, text generation tasks, evaluation,
challenges, systematic literature review

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation email; June 03–05, 2018; Woodstock, NY††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies††ccs: General and
reference Surveys and overviews††ccs: Computing methodologies Natural language
generation††ccs: Computing methodologies Natural language processing

##  1\. Introduction

A long-distant dream once only possible in science fiction movies is almost a
reality. Machines, or more particularly, artificial intelligence (AI) systems,
are becoming human companions to solve many cognitive problems and mental
routine tasks (Ouyang et al., 2022; Achiam et al., 2023; Touvron et al., 2023)
similar to how industrialization has automated numerous physical problems and
bodily routine activities. Producing language and text plays a vital role in
these advanced systems; a known factor of human intelligence that sets us
apart from other known living organisms is our capacity to communicate in
complex ways (Premack, 2004; Dennett, 2013). Language allows us to create
hypothetical scenarios, travel back and forth in time, reflect and think about
problems, and communicate these processes to others.

Major breakthroughs in AI have been achieved once models became capable of
modeling natural language, specifically with large language models, to
generate texts that are on par with human writing (Uchendu et al., 2021; Dou
et al., 2022; Wahle et al., 2022a). Together with advanced deep learning
architectures, large-scale data, and increasingly cheap computing
infrastructure, this has unveiled a new paradigm of training AI assistants at
a large scale. Anyone with access to the internet can now have their own AI
assistant to automate laborious and time-consuming tasks like drafting emails,
filling forms, or developing software. Although this utopian scenario would
strike most as a serendipity event, it is nothing more but the result of
constant, collaborative, and incremental efforts between researchers and
practitioners worldwide from AI, engineering, statistics, linguistics, and
natural language processing (NLP), inter alia, over the years.

From the early days of distributional semantics (Harris, 1954), rule-based
question answering systems (McKeown, 1982), the first neural probabilistic
language models (LMs) (Bengio et al., 2000), to GPT-4 (Achiam et al., 2023),
LLaMA (Touvron et al., 2023), and Gemini (Team et al., 2023), NLP has been a
key protagonist fueling the fast progress in AI. Nowadays, LMs solve problems
in a text-to-text manner, receiving and responding in convincing and credible
natural language. This problem-solving method is inspired by how humans solve
various problems by formulating answers as a sequence of words representing a
particular meaning, a process known as text generation. In general terms, text
generation is the process of creating a natural language text. In the early
days of text generation, models would use structured data, grammatical rules,
and templates to construct text (Winograd, 1972; McKeown, 1982). Today, most
approaches use neural networks to estimate the probability of the next word in
a sequence of words (Bengio et al., 2000).

Text generation solutions are highly versatile and powerful as they can
perform a myriad of tasks, such as generating stories (Brown et al., 2020) or
performing human-like reasoning (Lei et al., 2022). The wide range of
applications and marked interest in research and development detract from its
thorough understanding for those involved in funding, research, and product
development, which directly or indirectly impact society. Continuous
reflection and evaluation of proposed research models, tasks, datasets, and
open challenges are key for sustainable development and methods that
responsibly benefit humanity. Thus, systematic literature reviews and surveys
are crucial to condense and organize the available related work according to
special interests, retrospectively discuss these opportunities and risks, and
draw recommendations for future research.

In this paper, we provide an overview of recent text generation research
between January 2017 and August 2023, as it permeates most activities in NLP
concerned with text production (e.g., translation (Kocmi et al., 2023),
summarization (Liang et al., 2023)). Our systematic literature review has
three main focuses: tasks and sub-tasks (section 3), evaluation metrics
(section 4), and challenges (section 5). Specifically, we devise the following
key questions to organize our investigations:

  1. (1)

What constitutes the task of text generation? What are the main sub-tasks?

  2. (2)

How are text generation systems evaluated? What are the concomitant
limitations?

  3. (3)

What are the open challenges in text generation?

  4. (4)

What are prominent research directions in text generation?

Figure 1 gives an overview of the most prominent tasks and associated
challenges in text generation. We identify five main tasks: open-ended text
generation, summarization, translation, paraphrasing, and question answering
(Section 3). For each task, we review their relevant characteristics, sub-
tasks, and specific challenges. Next, we assess commonly employed evaluation
methodologies in the field (i.e., model-free and model-based metrics) and
discuss their limitations (Section 4). In addition, we also identify nine
prominent challenges common to all tasks and sub-tasks in recent text
generation publications: bias, reasoning, hallucinations, misuse, privacy,
interpretability, transparency, datasets, and computing (Section 5). Lastly,
we revisit, summarize, and answer our research questions (Section 6). For
reproducibility, we publicly share the details of our methodology (e.g., key
phrases, subjective decisions, exclusion criteria, code) and metadata of
considered publications in an open-access
repository111https://github.com/jonas-becker/text-generation/.

[A taxonomy of the main tasks and sub-tasks of text generation and their
respective challenges.](1) Open-Ended Text Generation: Open-Domain, Story
Generation, Dialogue Generation. Challenges: Bias, Factuality, Toxicity,
Coherence. (2) Summarization: Single-Document, Multi-Document, Dialogue
Summarization. Challenges: Faithfulness, Context, Discourse Phenomena. (3)
Translation: Sentence-Level, Document-Level. Challenges: Ambiguities, Low-
Resource Languages, Long-Sequences. (4) Paraphrasing: Uncontrolled,
Controlled. Challenges: Standalone References, Few Type Labels. (5) Question
Answering: Internal Knowledge, External Knowledge. Challenges: Reasoning, Non-
Answerability, Non-Factoid Questions. (6) Related Challenges: Bias, Misuse,
Reasoning, Hallucinations, Privacy, Transparency, Interpretability, Datasets,
Computing.

Figure 1. Taxonomy of the main tasks in text generation and their respective
challenges.

###  1.1. Related Literature Reviews

Since 2017, NLP has witnessed remarkable development with the potential to
impact the world substantially. More specifically, the field of text
generation has experienced a drastic increase in its number of publications
recently (Ramos et al., 2023). As a result, focused efforts exist to survey
and organize the available literature. Most of these efforts focus on
particular aspects of the problem, such as efficient Transformer architectures
(Tay et al., 2020; Lin et al., 2021), adaptations of pre-trained modes for
unconditional text generation (Li et al., 2021), the use of reinforcement
learning for sequence-to-sequence models (Keneshloo et al., 2019), the use of
external knowledge to improve the quality of generated text (Yu et al., 2022),
and specific challenges (e.g., hallucination (Ji et al., 2023), gender bias
(Sun et al., 2019), human evaluation (Clark et al., 2021)).

Only a few literature reviews provide a broad analysis of text generation.
Zhao et al. (2023) investigate the pre-training, fine-tuning, utilization, and
evaluation of recent LMs with more than ten billion parameters in selected
tasks. Fatima et al. (2022) survey 90 publications (2015-2021) on text
generation using deep neural networks and organize their approaches, quality
metrics, datasets, languages, and applications. Goyal et al. (2023) survey the
technical application, evaluation, and challenges of text generation from
after the year 2011.

Our systematic literature review contributes with new key aspects to the
existing ones. First, we include a large set of 244 resources resulting from
subsequent exploration and writing covering the literature until March 2024.
Second, different from other literature reviews, such as (Ji et al., 2023), we
select our main publications systematically. Rather than choosing specific
sub-fields and techniques (Li et al., 2022; Zhao et al., 2023), we adopt a
bottom-up approach and infer the most relevant sub-tasks, datasets, evaluation
procedures, and associated challenges in text generation. Finally, we dedicate
a specific analysis to open challenges in text generation that can explored in
the near future, helping to inform researchers in the field about which
research gaps are interesting to work on.

##  2\. Methodology

This systematic literature review is inspired by the guidelines of Kitchenham
and Charters (2007) and Okoli (2015) and comprises three parts: search and
retrieval, automatic filtering, and manual assessment. Figure 2 provides an
overview of the process. We provide the source code for reproducing the
literature retrieval on GitHub1.

[The pipeline of our systematic literature review.](1) Search results from
Semantic Scholar: 1669. We exclude 205 papers older than 2016. We select the
top five papers by influential citations per query and year. We exclude papers
until 2021 with less than 2 citations. (2) Titles and abstracts for relevance
annotation: 279. We exclude 143 irrelevant papers. (3) Papers selected for
full review: 136. We exclude 31 irrelevant papers. We add 143 additional
resources. (4) Papers included in this work: 244

Figure 2. Pipeline of this systematic review. Red-colored boxes show papers we
excluded; Green-colored boxes show papers we added.

###  2.1. Search and Retrieval

Query construction. Our search queries use primary and secondary terms to
select candidate papers. While primary terms aim to match papers in text
generation, secondary terms focus our query on specific characteristics (e.g.,
training). As the two primary terms, we use text generation and machine-
generated text. We generate the secondary terms by manually inspecting
existing field surveys and using suggestions made by ChatGPT-4222version May
12, 2023. We manually review the list of search terms, merge overlapping
terms, and conduct consensus voting among the authors for the final selection.
We consider the following 14 secondary terms: paraphrase, nlp, natural
language generation, language model, evaluation metrics, bias, privacy,
controllable, creative, machine, automated, task, training, cost, co2
emission, and detection.

Retrieval. We search Semantic Scholar for papers (including preprints from
arXiv) by building all permutations of primary and secondary terms as queries,
leaving us with 30 queries (e.g., “text generation evaluation metrics”). We
restrict our search to Semantic Scholar’s field of study to computer science
to exclude articles on text generation in other disciplines (e.g., text
generation tasks by humans in psychology studies)333According to Semantic
Scholar, the fields of study attribute has an accuracy of 86% and is based on
titles and abstracts: https://blog.allenai.org/announcing-s2fos-an-open-
source-academic-field-of-study-classifier-9d2f641949e5. We retrieve the top
100 papers per query and year, ranked by Semantic Scholar’s native ranking
algorithm444https://api.semanticscholar.org/api-docs/. Our search results in
1669 publications published between 1966—2023.

###  2.2. Automatic Filtering

Temporal criteria. The transformer architecture has disrupted text generation
and is a cornerstone of modern text generation development. As previously
discussed, many reviews focus on text generation before the transformer era or
its early days (e.g., up to 2021 (Fatima et al., 2022)) We consider
publications starting from the publication of Transformer (2017) (Vaswani et
al., 2017). Our initial collection, which consists of 1669 publications, is
reduced to 1464 papers when adding the temporal filter.

Citation criteria. Next, we filter the results using Semantic Scholar’s
influential citations (Valenzuela-Escarcega et al., 2015). An influential
citation traces key works that another work relies on, e.g., measured by
citations in specific paper locations, research overlap, or content similarity
work (Valenzuela-Escarcega et al., 2015). Influential citations, therefore,
provide a proxy for measuring a publication’s impact on other publications in
the field. To even out the distribution of papers regarding yearly coverage,
we sample the top five papers per query and year by influential citations,
yielding 337 papers. We select papers with more than two influential citations
until 2021 to estimate papers of high influence in text generation. We treat
recent works differently as older papers tend to have more citations than
recent ones, although both can be similarly impactful (Abdalla et al., 2023;
Wahle et al., 2023b). Thus, we relax our restrictions and do not remove papers
with less than two influential citations for the years 2022 and 2023. After
automatic filtering rules, we are left with 279 works.

###  2.3. Manual Assessment

Eligibility criteria. We manually assess the titles and abstracts of 279 works
from 2017 to 2023 to judge their relevance regarding text generation. We
remove papers that are too specific for the scope of this review (e.g.,
multimodality) and eliminate works that employ studies to another field of
study (e.g., medicine). We release a table of our relevance annotations to our
GitHub repository1. After testing papers for eligibility, we judge 136
publications as relevant.

| Prior | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023+ | Total
---|---|---|---|---|---|---|---|---|---
Search | 205 | 65 | 113 | 188 | 334 | 438 | 268 | 58 | 1669
Filtering | 0 | 47 | 64 | 48 | 48 | 41 | 41 | 48 | 337
Assessment | 37 | 12 | 19 | 27 | 39 | 29 | 30 | 55 | 244
Table 1.  Amount of papers per year after search and retrieval, during
automatic filtering, and after manual assessment.

Reading. We summarize each relevant paper, considering their main
contributions, applications, tasks, datasets, potential challenges, and
limitations. The publications are tagged with the different main aspects of
this review (e.g., dataset, metric) to facilitate their organization. To
complement our dataset with other relevant work, not in Semantic Scholar, we
add auxiliary papers to the systematic search, which we identify in three
ways: (1) they were referenced by other relevant works and contained the
source method that has been applied by that other work, i.e., giving credit to
the original idea, (2) by specifically exploring Google Scholar for subfields
of text generation from regular discussions between the authors and consulting
additional research-related experts from the field, or (3) by adding more
recent papers that the authors are made aware of either through social media
or other forms of scholarly notifications. In this process, we add 143
relevant papers, totaling 244 papers covered in this literature review.

##  3\. Text Generation Tasks

Text generation, also called natural language generation, is a computational
linguistic task that produces human-like text from a latent non-linguistic
representation of information (Mcdonald, 1980; Katz, 1980; Reiter and Dale,
1997). Constructing text similar to humans is central to improving the
accessibility of technology and information, fostering productivity in writing
tasks, and improving (cross-lingual) communication between people. Early
personal computers were hardly accessible to the wider society. Today, most
young people can quickly learn to use computers’ central features to access
information and boost their productivity. But if computers could communicate
similarly to humans, the barrier to their accessibility could be lowered even
more. Specifically, fragile communities or people with higher learning curves
toward technology, such as the elderly, could benefit from more accessible
technology and information. Even for creative tasks, productivity can be
increased by supporting users with generated templates or boilerplate source
code. Generating texts can also automate routine mental tasks, like writing
standardized reports or summarizing data from documents.

However, text generation also has potentially harmful applications. For
example, Zellers et al. (2019b) describe a method for fake news generation in
which their model generates a plausible article for any headline. If these
bots can automatically generate convincing text, they could be misused to
manipulate public opinion, influence elections, and generate other harmful
content. Whether text generation is considered an opportunity, a risk, or
both, understanding the recent developments in this field is of the utmost
importance. Therefore, we define the problem of text generation, its most
prominent sub-tasks, concomitant characteristics, and open challenges.

###  3.1. Problem

Katz (1980) was one of the first to explore text generation as a process of
grouping different feature representations for generating kernel sentences,
deciding on transformations based on syntax and themes, and then combining
these kernels with appropriate modifications and pronoun adjustments to
produce cohesive text. Later, Reiter and Dale (1997) expanded the notion of
text generation to signals other than text, such as weather forecast data from
graphical weather maps (Goldberg et al., 1994) or the creation of technical
documentation via a knowledge base (REITER et al., 1995). Modern definitions
advocate for a process that leverages knowledge in computational linguistics
and artificial intelligence to automatically generate natural, cohesive, and
plausible texts within defined requirements (Li et al., 2022). In our
literature review, we delimit text generation as the synthetic production of
contextually relevant content in written natural language from an underlying
non-linguistic representation. While the input to a text generation system
during training might be a natural language, it is not directly converted to a
textual output. The output is derived from an underlying non-linguistic
representation (e.g., embeddings) displaying meaning.

At the time of writing, text generation is often performed via LMs (Li et al.,
2022; Brown et al., 2020). LMs provide a probabilistic description of natural
language and maximize the probability of the generated sequence, producing the
most probable text relevant to the context (Bengio et al., 2000; Jurafsky and
Martin, 2024). Language modeling and text generation are distinct yet
interconnected fields. While text generation focuses on producing content,
language modeling leverages the probabilistic structure of the language used
in the process. However, text generation does not necessarily have to employ
LMs. Examples are rule-based approaches from Katz (1980) or non-autoregressive
methods Xiao et al. (2022).

Our literature review identifies five most prominent areas related to text
generation, namely open-ended text generation, summarization, translation,
paraphrasing, and question answering (Figure 1). Open-ended text generation is
a task where newly generated text is iteratively conditioned on the previous
context so that the final output appears coherent and fluent (Li et al.,
2023b). Summarization is the generation of a text from one or more texts
conveying information in a shorter format (Radev et al., 2002). Translation
means converting a source text in language A to a target language B (Lewis et
al., 2020). Paraphrasing is the task of generating text that has
(approximately) identical meaning but uses different words or structures
(Dolan and Brockett, 2005; Vila et al., 2015; Wahle et al., 2023a). Question
answering takes a question as input text and outputs a streamlined answer or a
list of possible answers based on background knowledge (Cai et al., 2020). We
investigate these areas according to their specific sub-tasks and challenges.
We mention relevant datasets selectively but point to our GitHub repository
for a more elaborate list of available corpora1.

###  3.2. Open-Ended Text Generation

Task definition. Open-Ended Text Generation (OETG) generates new text based on
the previous context (prompt) so that the final output appears coherent and
fluent (Li et al., 2023b). Applications of OETG range from open-domain
response generation by AI assistants (Achiam et al., 2023) to complex story
generation (Liang et al., 2023).

Main characteristics. Compared to other text generation tasks like
summarization, OETG provides more freedom during content generation due to the
highly variable output length. For this, LMs that maximize the probability
distribution over word sequences are used to generate fluent text. Conversely,
human language mainly encompasses semantic and pragmatic characteristics
beyond grammatical and syntactic rules (Gehrmann et al., 2019). Nevertheless,
the fluency of machine-generated text can mislead human judgments on the
authorship of a text (Adelani et al., 2020).

Challenges. Since the rise of LMs (Brown et al., 2020; Devlin et al., 2019)
and their capacity to produce fluent texts, other desired features emerged for
OETG. As sequences generated by OETG systems can grow large, the risk of
producing logical inconsistencies increases. This makes coherence a key
challenge to solve in OETG (Guan et al., 2021; Liang et al., 2023). In
addition, due to their dependency on large training corpora, recent LMs are
subject to bias, prejudice, and toxicity from human language. Thus,
controlling attributes like sentiment (Kawamae, 2023; Adelani et al., 2020;
Liu et al., 2021) during OETG is an active research topic (Yang et al., 2023).

####  3.2.1. Sub-tasks and Datasets in Open-Ended Text Generation

We identify three popular sub-tasks of OETG, namely open-domain OETG, story
generation, and dialogue generation. Other sub-tasks like review generation
(Costa et al., 2018; Ni and McAuley, 2018), data-to-document generation (Lu et
al., 2023), and recipe generation (H. Lee et al., 2020) often use open-domain
OETG methods because those generalize well to said problems.

Open-domain. OETG is called open-domain if a model does not operate within a
specific topic or structure and can produce highly variable text in a myriad
of scenarios. Contemporary LMs, e.g., GPT (Achiam et al., 2023; Brown et al.,
2020; Radford et al., 2019) or LLaMA (Touvron et al., 2023), use prompts to
generate text that can span several topics, domains, or textual output
formats. The open-domain capabilities of recent LMs are achieved by pre-
training them on large diverse data (e.g., web text, knowledge bases (Yu et
al., 2022)) and tasks (e.g., instruction following (Ouyang et al., 2022),
question generation (Mulla and Gharpure, 2023)) Modern LMs rely on large
amounts of data scraped from web pages in their training objective, e.g.,
Common Crawl555https://commoncrawl.org/ (Brown et al., 2020; Touvron et al.,
2023). The number and variety of domains covered by these data sources are the
root cause of LMs generalizing across domains.

Challenges of open-domain OETG. Technology companies often propose the most
capable open-domain models as they have the resources to train them on large
amounts of data (Abdalla et al., 2023). Many of these models are behind closed
walls (Liu et al., 2023b). This makes the creation and reproducibility of
modern LMs particularly challenging, as many companies tend not to disclose
technical details about their models, such as used data or hyperparameters
(Achiam et al., 2023). Few companies follow an open-source approach when it
comes to LM development (Touvron et al., 2023).

Story generation. A story generation model creates a narrative text that
appears coherent to the reader (Guan et al., 2021; Liang et al., 2023). This
sub-task employs the description of consecutive narrative events, supplemented
with characters that follow a plot consistently. We find that many approaches
for story generation employ additional modules to improve the consistency of
the generated text. Fan et al. (2018) use a convolutional LM to generate a
specially designed prompt and enhance the consistency of story generation with
a text-to-text model. Planning the generation process in a hierarchical way
can enhance creativity (Peng, 2022). Yang et al. (2022b) employ a planning
module that produces a story premise with a setting, characters, and outline.
Additionally, they use a dedicated module to make local edits so that the
generated story appears consistent. The WritingPrompts dataset (Fan et al.,
2018) includes 300k stories from Reddit paired with human-written prompts.

Challenges of story generation. The production of long and coherent stories
with a consistent narrative and developing characters (i.e., unique and
changing properties over the temporal shift in the story) is one of the main
challenges in story generation (Alabdulkarim et al., 2021). Commonly employed
decoding algorithms like beam search produce incoherent text for story
generation (Wiher et al., 2022). Lu et al. (2022b) propose a decoding
algorithm with lookahead heuristics that enables foresight planning.

Dialogue generation. The task to simulate verbal or written communication
between two or more parties is called dialogue generation (Zhang et al.,
2020c; Li and Zhao, 2023). While a two-party setup (i.e., training data and
architecture) is often seen in AI assistants, a multi-party one is more common
during, e.g., movie dialogue. Bidirectional LMs (e.g., BERT (Devlin et al.,
2019)) and autoregressive LMs (e.g., GPT (Achiam et al., 2023; Brown et al.,
2020; Radford et al., 2019)) are not directly optimized on dialogue datasets
as they are trained on general text (e.g., Wikipedia articles). The one-to-
many relationships in dialogue (Bao et al., 2020) are a reason to avoid fine-
tuning such LMs on conversational data directly. Thus, unidirectional dialogue
requires a training mode different from bidirectional models. Bao et al.
(2020) and Zhang et al. (2020b) do a workaround by further pre-training the LM
on conversational data. They integrate uni- and bidirectional flexible
processing and model the one-to-many relationships with a latent variable. For
pre-training and testing conversational models, mostly synthetic data exists.
For example, the AMI corpus (Mccowan et al., 2005) consists of 279 hours of
textualized synthetic meeting dialogue.

Challenges of dialogue generation. Both two- and multi-party dialogue
generation are inherently challenged by semantics, consistency, and
interactivity when applied to open-domain problems (Huang et al., 2019). Huang
et al. (2019) identify that these challenges are influenced and addressed by
long context, the personality of characters, sentiment, and dialogue policies.
While research on two-party dialogue is vast, multi-party dialogue generation
remains underexplored (Kann et al., 2022), possibly because natural multi-
party dialogue often occurs in confidential scenarios like business meetings.
Investigating the dynamics of multi-party dialogue could provide an
understanding of the dynamics behind turn-taking (Skantze, 2021) and multi-
agent interaction (Wang et al., 2023b; Jinxin et al., 2023).

###  3.3. Summarization

Task definition. Summarization describes the generation of a short text using
one or more references (Radev et al., 2002). These references are usually more
than double the length of the produced summaries (Radev et al., 2002). Many
works differentiate between extractive and abstractive summarization (Fabbri
et al., 2021; Bražinskas et al., 2020; Taunk et al., 2023). Extractive
summarization concatenates spans of the input text that must be summarized to
produce the final output text (Moratanch and Chitrakala, 2017). Abstractive
summarization generates new phrases and sentences that can differ from the
original text (Zhang et al., 2020d). Hence, abstractive summarization can
generate text with low lexical overlap to the source, while extractive
summarization always has overlap regarding the extracted text spans (Paulus et
al., 2018; Zhang et al., 2020d).

Main characteristics. Extractive systems use statistics to identify relevant
text and are usually simple to implement, e.g., by fine-tuning an LM (Liu et
al., 2020) or ranking sentences (Xiao and Carenini, 2019). Abstractive systems
conversely use semantic features like word embeddings (Zhang et al., 2020d) to
determine the meaning of texts and summarize them accordingly. In recent
years, abstractive summarization has seen a surge of techniques due to its
ability to generate lexically independent summaries (Wang et al., 2018; Zhang
et al., 2019). Some efforts combine both extractive and abstractive methods
for summarization to address the long computing times that would come with
purely abstractive approaches for long documents (Taunk et al., 2023).

Challenges. Extractive summarization often produces duplicated information and
inconsistent summaries sensitive to the statistical features of the source
text (Munot and S. Govilkar, 2014) while abstractive methods suffer from large
contexts when summarizing large documents (Gambhir and Gupta, 2017). In
addition, abstractive systems lack faithfulness and tend to hallucinate
(Maynez et al., 2020).

####  3.3.1. Sub-tasks and Datasets in Summarization

According to our investigations, the most popular summarization sub-tasks are
single-document summarization, multi-document summarization, and dialogue
summarization.

Single-document summarization (SDS). In SDS, one generates an accurate and
concise summary from a single input document, e.g., a news article (Zhang et
al., 2019, 2020d). Extractive SDS can be executed by employing bidirectional
masked LMs like BERT (Liu and Lapata, 2019). Copy mechanisms employed by a
pointer-generator network are used to leverage relevant tokens from the source
as part of the output (Kumar et al., 2019). A decoder equipped with a pointer-
generator network makes a soft choice at each decoding step between copying
from the source and generating from the vocabulary (Zhang et al., 2019). This
way, the encoder can predict out-of-vocabulary words by selecting appropriate
tokens from the source text. Xiao and Carenini (2019) show that separately
taking local and global context into account can improve the quality of
summaries.

Most SDS datasets we identify originate from the news domain (Narayan et al.,
2018; Nallapati et al., 2016). This reliance is possibly caused by the fact
that these datasets can conveniently leverage headlines or first sentences
from news websites as summaries (Narayan et al., 2018) and thus, are
comparatively easy to create. For example, the XSum dataset (Narayan et al.,
2018) includes 227k single-sentence summaries of BBC articles. The CNN/Daily
Mail dataset (Nallapati et al., 2016) is a corpus with 312k multi-sentence
summaries obtained by concatenating human summary bullets.

Challenges of SDS. Using transformer-based architectures for long documents is
challenging because of the quadratic time and memory complexity (Vaswani et
al., 2017). This limitation is mitigated by specialized architectures like
Longformer (Beltagy et al., 2020), which employ a linearly scaling attention
mechanism. Besides this, summarization systems often lack faithfulness to the
source text and tend to hallucinate information (Maynez et al., 2020).

Multi-document summarization (MDS). MDS summarizes a set of multiple topic-
related documents to generate a single executive summary (Ma et al., 2023).
Maynez et al. (2023) identify several strategies for architecture design,
ranging from ensemble networks or hierarchical approaches to graph neural
networks or fine-tuning of pre-trained LMs. Liu et al. (2019) represent cross-
document relationships via an attention mechanism. The Multi-News dataset
(Fabbri et al., 2019) provides 46k news articles with an average number of 3.5
documents per summarized cluster.

Challenges of MDS. MDS suffers from similar challenges as SDS but comes with
an additional set of issues. Cross-document relations and conflicting-,
duplicate-, and complementary information increase the challenges in mapping
similar entities, events, and other related information (Ma et al., 2023;
Gambhir and Gupta, 2017). We also find that the diversity of available
datasets is low, possibly due to the expenses needed to create high-quality
MDS datasets.

Dialogue summarization. A dialog summary is produced by leveraging a multi-
turn conversation as input. This sub-task is also important for meeting
summarization (Feng et al., 2021), as one can reduce costs and time on manual
note-taking activities. Zhu et al. (2020) capture the dialogue structure by a
hierarchical approach. They employ a word-level Transformer that processes the
sequence of a single turn, while a turn-level Transformer processes the
information of all turns in a meeting. Feng et al. (2021) leverage a
relational graph encoder to model discourse relations. Most datasets for
dialogue summarization consist of synthetic data. The AMI corpus (Mccowan et
al., 2005) comprises 279 hours of synthetic meeting dialogue textualized from
speech with annotations of extractive and abstractive summaries. Its
participants play different roles in a team responsible for hardware design.

Challenges of dialogue summarization. Current summarization methods are
particularly challenged by discourse phenomena like coreference errors (e.g.,
ambiguous pronouns) or discourse link errors (e.g., temporal ordering)
(Pagnoni et al., 2021). We identify that datasets in dialogue and meeting
summarization are scarce, especially when looking for non-synthetic data. We
suppose that this is because of confidentiality issues between the
participants and discussed topics. Training and evaluation on synthetic data
could cause unexpected problems, such as a performance drop in real
conversations because naturally occurring dialogue might differ structurally
or linguistically.

###  3.4. Translation

Task definition. Translation, also referred to as machine translation, is a
task in which a (textual) source in language A is converted into a target
language B (Lewis et al., 2020).

Main characteristics. Translation can involve different modalities, e.g.,
text-to-code translation (Shi et al., 2022). The most popular task, however,
is text-to-text, and all publications of our search concern text-to-text
problems (e.g., translation of German to English text).

Challenges. One of the main challenges in translation is to guarantee that the
generated translation conveys the same meaning as its source (i.e., avoiding
semantic shift). Most contributions in translation often focus on high-
resource languages (e.g., English and Chinese) because of the abundant
resources available. Consequently, training and evaluation are more difficult
for low-resource languages due to the lack of accessible data covering them
(Taunk et al., 2023). The mismatch between train and test data can
significantly impact the performance of any translation system (Currey et al.,
2020; Artetxe et al., 2023). To mitigate this problem, Artetxe et al. (2023)
adapt the human training data by back-translation (e.g., Portoguese to English
to Portoguese), making it more similar to a machine-generated model input
during test time. They assume that applying machine translation twice induces
a similar error and thus reduces the gap between train and test data. A well-
selected dataset can also help reduce the train/test mismatch (Vilar et al.,
2022).

####  3.4.1. Sub-tasks and Datasets in Translation

According to our investigations, the most popular translation sub-tasks are
sentence-level and document-level translation. Most datasets in translation
consider short sequences for their tasks (sentence-level) as they are easier
to annotate than longer sequences (document-level). Thus, current systems are
often more capable of handling short sequences as they are largely trained on
corresponding data. However, document-level information might help to solve
word ambiguity as the longer context can allow better handling of discourse
phenomena like coreference, omissions, and coherence (Zhang and Zong, 2020).
Nevertheless, document-level translation contributions are still incipient.

Sentence-level. A single sentence is translated from a source to a target
language for the sentence-level translation task. Recent works propose a
variant of the encoder-decoder Transformer, predicting tokens with a
classifier over a large database of translation examples (Khandelwal et al.,
2021), or prompt existing pre-trained LMs (Zhang et al., 2023) to perform
machine translation.

Sentence-level translation involves sentence-aligned datasets that usually are
sampled from a full-sized document. The most popular datasets in the
translation field come from the WMT General Machine Translation task (Kocmi et
al., 2023). Since 2008, there have been regular releases and updates of the
WMT dataset composed of several other corpora (Callison-Burch et al., 2008;
Kocmi et al., 2023). Each corpus covers a different set of languages (e.g.,
Chinese, German, Japanese) and domains (e.g., e-commerce, conversation, news),
with a few thousand examples per language.

Challenges of sentence-level translation. Sentence-level translation is
specifically limited due to a lack of context, for example when referring to
out-of-context personalities by pronouns or when encountering words that can
have multiple translations depending on the language (Zhang and Zong, 2020).
Word ambiguities are especially difficult to handle when only a single
sentence is provided as a source. In addition, translation models trained on
short sequences perform significantly worse when tested on long sequences (60+
words) (Koehn and Knowles, 2017). Thus, success in semantic preservation for
longer sequences is especially influenced by the variability, context, and
ambiguity of different languages.

Document-level. Translation on the document-level transforms multiple
sentences and paragraphs into another language by taking the sentences’ inter-
relations into account (Maruf et al., 2022). Hence, document-level translation
systems leverage in-document context to mitigate ambiguities that are
difficult to handle with single sentences. The exact length of the segmented
sequences heavily differs between works, ranging from small paragraphs to
complete books. Thus, we reference efforts to translate text into sequences
longer than sentence-level as document-level translation. A recent approach by
Wang et al. (2023a) leverages the long context modeling abilities of recent
LMs.

Data for document-level translation is scarce. Our findings suggest that the
lack of datasets in this sub-task leads to high upfront costs for conducting
research in document-level translation because it requires creating high-
quality translations of longer documents. Many of the available datasets are
paid resources that require a fee to access such as the NIST corpora (NIST
Multimodal Information Group, 2010, 2013). They are relatively small (100
samples per language) but provide reference translations for longer news
articles. NIST datasets also only cover a few languages and are not uniformly
structured. For example, NIST 2005 (NIST Multimodal Information Group, 2010)
contains 4-way reference translations of Newswire articles in Chinese and
Arabic (100 samples per language), while NIST 2012 (NIST Multimodal
Information Group, 2013) contains 222 documents with Chinese-to-English pairs
from Newswire and web data.

Challenges of document-level translation. The lack of research in document-
level translation also reflects the difficulty of this task. Discourse
phenomena like coreference, omissions, and coherence can occur that are
special for long sequences, facilitating the performance differences between
translating shorter or longer sequences (Zhang and Zong, 2020). While
classical single-sentence translation models underperform in long sequence
scenarios, modern LMs can handle more context at the cost of their
interpretability (Wang et al., 2023a).

###  3.5. Paraphrasing

Task definition. Paraphrases are texts expressing (approximately) identical
meanings that use different words or structures (Dolan and Brockett, 2005;
Vila et al., 2015; Wahle et al., 2023a). Thus, paraphrasing aims towards the
generated text having a high semantic similarity to the source text.

Main characteristics. In paraphrasing, the output length is not a relevant
criterion which is different from other tasks like translation. Thus, a
paraphrased text can be the same length, shorter, or longer than the original
text. Paraphrases can be generated in several ways, such as by replacing or
adding words, making grammatical changes, or changing the order of words
(Kovatchev et al., 2018).

Challenges. A key challenge of paraphrasing is the lack of suitable evaluation
metrics. The similarity of paraphrases is difficult to measure, with different
lexical or semantic changes taking place. Most authors use BLEU or other
n-gram-based metrics to test their systems’ performance (Grusky, 2023). As
Shen et al. (2022b) point out, this has issues because their correlation with
human judgments is low, and there are many ways to paraphrase a text.

Consequently, multiple references for each data point would facilitate the
training of paraphrasing systems (Zheng et al., 2018), but we find that most
corpora still only include one reference (Kovatchev et al., 2018; Wang et al.,
2017). Becker et al. (2023) point out the difference between human- and
machine-generated paraphrases, suggesting the latter are semantically closer
to their original source text when compared to human paraphrases. They also
identify some corpora as thematically balanced while others show concentrated
or unbalanced coverage of domains like politics.

####  3.5.1. Sub-tasks and Datasets in Paraphrasing

We identify two major sub-fields of paraphrasing research, namely uncontrolled
paraphrasing and controlled paraphrasing. As uncontrolled paraphrasing does
not come with additional constraints on the generation process, it is
inherently characterized by its output variability. The other branch of
paraphrasing studies aims to control the variability of paraphrases during
generation. As recent models show satisfactory capabilities on uncontrolled
paraphrasing tasks, the focus has shifted towards developing systems that
provide more control to the user. Thus, uncontrolled paraphrasing is well-
established compared to controlled paraphrasing which has seen a surge in
recent years.

Uncontrolled. When only the notion of a text being paraphrased into another is
used without caring for the specific changes that take place, we call this
task uncontrolled paraphrasing. Approaches use an encoder-decoder architecture
that first creates a semantic representation of the input text and then
outputs a paraphrase (Egonmwan and Chali, 2019). Garg et al. (2021) fine-tune
an autoregressive LM to generate paraphrases. Several datasets are available
for paraphrasing considering uncontrolled scenarios. For example, TURL (Lan et
al., 2017) comprises 2.8m multi-rater annotated sentence pairs from Twitter
news. Quora Question Pairs (QQP) (Wang et al., 2017) comprises 400k similar
questions from the Quora forum with binary paraphrase annotations.

Challenges of uncontrolled paraphrasing. Since many possible paraphrases can
be produced, the n-gram-based comparison to a single reference is
questionable. To encourage diversity in paraphrases, we, therefore, suggest
the use of multiple references per source text (Lan et al., 2017; Zheng et
al., 2018). However, these are often not provided in today’s paraphrasing
datasets. We suggest that new corpora should come with multiple reference
paraphrases as a standard.

Controlled. Text can be paraphrased in a controlled scenario through different
changes (e.g., structure, morphology). Subfields of research propose models
that generate paraphrases of specific types like polarity change (Wahle et
al., 2023a). Others try to achieve controllable characteristics like quality
(Bandel et al., 2022) or have their model follow structural constraints like
syntax (Chen et al., 2019; Kumar et al., 2020). Paraphrase types were
initially categorized by Vila et al. (2014). Later works (Kovatchev et al.,
2018) expand this idea and cover morphology-based, lexicon-based, lexico-
syntactic-based, syntax-based, discourse-based, and other changes to a source
input. Wahle et al. (2023a) show that paraphrases of a specific type can be
generated using autoregressive models like BART (Lewis et al., 2020) or
PEGASUS (Zhang et al., 2020d). Chen et al. (2019) and Kumar et al. (2020) use
semantic and syntactic encoders for syntax-controllable paraphrase generation.
A Transformer encoding a constituency tree can capture parent-child and
sibling relations, producing outputs constrained on a syntactic template (Yang
et al., 2022a). Bandel et al. (2022) directly control the quality of a
generated paraphrase by taking a vector of semantic similarity and the
syntactic and lexical distance as a quality constraint. Their approach allows
the model to choose which attributes to favor during generation to achieve the
desired quality. Zhang et al. (2021) leverage manifold sentiment information
to paraphrase a text.

The ETPC dataset (Kovatchev et al., 2018) comes with the extended paraphrase
typology. This includes granular annotations of the categories morphology-
based, lexicon-based, lexico-syntactic-based, syntax-based, discourse-based,
and other changes. Though it only includes 5k samples and shows a balanced
distribution of covered topics (Becker et al., 2023), other datasets usually
lack such granular type annotations. However, some other datasets focus on one
specific paraphrase type like sentence splitting (Botha et al., 2018) or
entailments (Williams et al., 2018). Overall, we see that datasets are
abundant for specific paraphrase types like sentence splitting, while other
aspects like polarity changes, syntax control, or sentiment control require
more specific datasets to be released or the adaption of non-paraphrasing
research on controllability (Ghosh et al., 2017; Huang et al., 2020; Yang et
al., 2023; Seo et al., 2023; Yang and Klein, 2021).

Challenges of controlled paraphrasing. While uncontrolled paraphrasing is a
well-established task, we find that controlled paraphrasing tasks such as
paraphrase type generation are incipient (Wahle et al., 2023a). Thus, many
architectural characteristics that make paraphrasing systems superior remain
unclear as they also depend on the controlled attribute. We could not identify
many datasets that provide a comprehensive selection of annotated paraphrase
types or controlled characteristics like sentiment. Therefore, we encourage
other researchers from the field to construct comprehensive datasets that
derive paraphrase-type labels similar to the ETPC dataset (Kovatchev et al.,
2018). This would foster additional research on controlled paraphrasing in the
future.

###  3.6. Question Answering

Task definition. Question answering (QA) takes a question as input and outputs
a streamlined answer or a list of possible answers based on background
knowledge (Cai et al., 2020). This background knowledge can either be provided
as a supplementary document, a knowledge base, or leveraged from training data
when no additional context is provided.

Main characteristics. Background knowledge of a QA system can be internal or
external (Yu et al., 2022). Internal knowledge takes place within the training
data and input text(s), including but not limited to keywords, topics,
linguistic features, and internal graph structure. External knowledge
acquisition occurs when knowledge is provided from outside sources, such as a
knowledge base, knowledge graph, or grounded text. While there exist QA
systems (Cai et al., 2020) for topic-restricted domains (e.g., math QA), most
research focuses on open-domain QA, i.e., QA systems with capabilities that
are not constrained on a topic because they generalize better to practical
problems.

Challenges. Evaluation of generated answers is increasingly difficult for more
abstractive settings like open-domain QA where the source does not directly
indicate what the correct answer might be (Durmus et al., 2020). To bridge the
gap of interpretability, researchers draw inspiration from how humans
synthesize a chain of thought for reasoning. Although Chain-of-Thought
Prompting (Wei et al., 2022) and Solo Performance Prompting (Wang et al.,
2023b) are competent approaches to improve reasoning in QA systems, their
implementation is not trivial because they require a specific prompt design
and large models that are capable of eliciting reasoning this way.

####  3.6.1. Sub-tasks and Datasets in Question and Answering

According to our investigations, the most popular QA sub-tasks are internal
knowledge-grounded QA (e.g., open-domain QA) and external knowledge-grounded
QA (e.g., reading comprehension).

Internal knowledge. If a model can not leverage any additional resources to
answer a question, i.e., without supplemental knowledge at inference time, the
QA task is grounded in internal knowledge (Yu et al., 2022). The system must
produce an answer by leveraging internal knowledge learned from training data
and the input question only. Corresponding systems can be employed with pre-
trained LMs that utilize the internal knowledge memorized from web data (Tan
et al., 2023).

Large and diverse pre-training corpora like Common
Crawl666https://commoncrawl.org/ increase the amount of internal knowledge
that can be leveraged for QA which is especially relevant for open-domain
scenarios. More specific datasets are used to fine-tune a model to perform
better in specific domains and to evaluate QA systems.

Challenges of internal knowledge-grounded QA. A core challenge that comes with
QA systems is their tendency to hallucinate information (Zhuang et al., 2023),
a characteristic that heavily countervails the goal of factual QA. Thus, there
exist many efforts in mitigating hallucinations, making LMs more reliable for
QA tasks (Zhou et al., 2021; Ji et al., 2023). We discuss hallucinations
thoroughly in Section 5.3.

External knowledge. QA can involve additional inputs (external knowledge) to
the model at inference time, e.g., a knowledge base, a knowledge graph, or
grounded text. The task that uses grounded text as external knowledge is also
known under the term reading comprehension. For this, LMs with sufficient
context-modeling capabilities can be prompted to answer a question based on a
provided source text (Tan et al., 2023). Nishida et al. (2020) improve the
out-of-domain performance of their QA system by stacking a reading
comprehension layer and an LM layer on top of BERT (Devlin et al., 2019).
Zhuang et al. (2023) employ 13 external tools, like a code interpreter or a
mathematical computer, to enhance the performance of a large LM on the QA
task.

QA datasets are abundant, especially in the subfield of reading comprehension
(Rogers et al., 2023). The widely used SQuAD dataset (Rajpurkar et al., 2016)
contains crowdsourced reading comprehension questions with reference answers
supported by a paragraph. SQuAD 2.0 (Rajpurkar et al., 2018) adds 50k
unanswerable questions to the base dataset that challenge models to predict
whether the generation of an answer is feasible at all. Du et al. (2017) and
Yuan et al. (2017) provide a model that can generate synthetic reading
comprehension questions. Gao et al. (2019) propose an advanced system with
controllable difficulty levels. It can be used to generate more samples on
specific domains if crowdsourcing is not an option.

Challenges of external knowledge-grounded QA. Many recent LMs still cannot
perform more complex reasoning during QA at a satisfactory level (Tan et al.,
2023), making this an active research topic. More challenges arise with non-
answerability or non-factoid questions as LMs are usually trained to always
generate some text as the true answer (Rajpurkar et al., 2018). Ideally, a
model should be able to tell the user when a competent answer seems
unrealistic, given the provided context. QA datasets like SQuAD (Rajpurkar et
al., 2016) have been criticized because they are overly dependent on the
similarity of question/answer sentences rather than on human-type reasoning,
meaning they require only superficial reading skills (Uto et al., 2023).
Datasets like ELI5 (Fan et al., 2019) can provide alternatives that require
more human-like reasoning and multi-step thinking.

##  4\. Evaluation

Several metrics have been proposed or adopted from other research fields to
evaluate, compare, and discuss text generation systems. These metrics are used
as proxy measures for desired characteristics of text generation, such as
coherence, fluency, and semantic diversity. Table 2 shows the summary of all
metrics identified. We identify 16 metrics from our 136 papers and organize
them into two major types: model-free and model-based evaluation. Model-free
metrics are usually based on static and rule-based algorithms like word
overlap (e.g., BLEU (Papineni et al., 2002)). Model-based metrics depend on
the embeddings of pre-trained LMs or word embeddings to produce similarity
scores (e.g., BERTScore (Zhang et al., 2020a)). Another field of evaluation
concerns human evaluation, which is deemed the gold standard to measure
characteristics like fluency, faithfulness, and coherence in text generation
systems (Ghosh et al., 2017; Ma et al., 2018; Qader et al., 2018; Feng et al.,
2021; Lu et al., 2022b). In this written work, we focus on automated metrics
only. For a full list including human evaluation procedures, we point to our
GitHub repository1.

Type | Category | Metric |  Description | Used
---|---|---|---|---
Model-free | N-gram |  BLEU (Papineni et al., 2002) |  Textual overlap between source and reference (precision). | 69
|  |  ROUGE (Lin, 2004) |  Textual overlap between source and reference (recall). | 46
|  |  METEOR (Banerjee and Lavie, 2005) |  Textual overlap between source and reference (precision and recall). | 32
|  |  CIDEr (Vedantam et al., 2015) |  Measures consensus on multiple reference texts. | 15
|  |  chrF++ (Popović, 2017) |  Character-based F-score computed using n-grams. | 13
|  |  Dist-n (Li et al., 2016) |  Measures generation diversity by the percentage of distinct n-grams. | 8
|  |  NIST (Doddington, 2002) |  Alters BLEU to also consider n-gram informativeness. | 6
|  |  Self-BLEU (Zhu et al., 2018) |  Measures generation diversity by calculating BLEU between generated samples. | 2
| Statistical |  Perplexity (Liang et al., 2023) |  Fluency metric based on the likelihood of word sequences. | 23
|  |  Word Error Rate (Huang et al., 2018) |  The rate of words that are different from a reference sequence based on the Levenshtein distance. | 11
| Graph |  SPICE (Anderson et al., 2016) |  Measures the semantic similarity of two texts by the distance of their scene graphs. | 6
Model-based | Hybrid |  BERTScore (Zhang et al., 2020a) |  Contextual token similarity to measure textual overlap. | 13
|  |  MoverScore (Zhao et al., 2019) |  Uses contextualized embeddings and captures both intersection and deviation from the reference for a similarity score. | 6
|  |  Word Mover Distance (Kusner et al., 2015) |  Distance metric to measure the dissimilarity of two texts. | 2
| Trained |  BLEURT (Sellam et al., 2020) |  Models human judgement on text quality. | 4
|  |  BARTScore (Yuan et al., 2021) |  Promptable metric that models human judgments on faithfulness besides precision and recall. | 3
Table 2.  Automated evaluation metrics in the text generation field. ”Used”
marks the number of papers that consider the metric in their publication from
our 136 filtered Semantic Scholar documents (proposing, surveying, or
applying).

###  4.1. Desired Characteristics in Text Generation

All metrics are used to estimate the desired characteristics we look for in
generated texts. We identify several characteristics like fluency and
coherence among our considered papers. The desired properties that constitute
text generation are vast but one can comprise these manifold aspects by the
general term quality, which we elaborate on in Section 4.1.1. Other
characteristics like bias, faithfulness, or reasoning also come with their own
evaluation procedures, but they are covered more thoroughly in Section 5.

####  4.1.1. Quality.

A high-quality text has to fulfill various criteria, such as having a correct
grammatical structure or being coherent and fluent. Sonntag (2004) decomposes
textual quality into three parts. First, intrinsic quality includes the need
for a text to be accurate, unbiased, and generated from a data source with a
high reputation. Secondly, contextual quality comprises the amount of text
being produced, the completeness of the output, and the relevance of the text
to the prompt or document (Deng et al., 2021). Timeline events should be
portrayed correctly and coherently. Third, representational quality requires a
consistent and concise output that can be easily understood. This also
involves the fluency of the output, and the text should be interpretable by
the reader. Note that this threefold definition of text quality is general,
potentially missing task-specific characteristics. For example, some tasks
like dialogue generation (Li and Zhao, 2023; Zhang et al., 2020c) specifically
require diverse outputs to avoid repetitions and encourage creativity (Chung
et al., 2023; Alihosseini et al., 2019).

Challenges of quality evaluation. Considering these dimensions of text
quality, evaluation in text generation research is commonly performed by
assessing the similarity of a machine-generated text to a reference text
written by humans (Papineni et al., 2002; Lin, 2004). To consider more
specific dimensions of quality or even additional attributes like diversity,
supplementary metrics need to be employed (Xu et al., 2017; Jagfeld et al.,
2018). Unlike traditional classification and regression tasks, where error
rates can be clearly defined, generative tasks have a complex answer space
and, in many cases, are subjective. A text can be more creative but less
precise or more lexically diverse but less semantically similar (Wahle et al.,
2023a). Most datasets and tasks (e.g., the summarization dataset XSum (Narayan
et al., 2018) or the paraphrasing dataset ETPC (Kovatchev et al., 2018)) only
have one summary as the reference. This lack of variation leads to overlapping
metrics rating several good outputs poorly. Thus, popular n-gram-based metrics
can provide scores that do not always align with human judgments (Jagfeld et
al., 2018). Moreover, n-gram-based metrics rely on hard or soft subsequence
alignments, which do not measure the interdependence between consecutive
sentences well (Fabbri et al., 2021) and show an unsatisfying correlation with
human judgments (Gatt and Krahmer, 2018). We outline the metrics aiming to
measure the degree of textual quality and other attributes to understand the
limitations of current evaluation methodologies.

###  4.2. Model-Free

The most used evaluation methods for text generation are model-free metrics
because they are easy to adapt and have low computational costs (Table 2). The
most common metrics used for evaluating the quality of machine-generated text
rely on n-gram-based comparisons, such as BLEU (Papineni et al., 2002), ROUGE
(Lin, 2004), and METEOR (Banerjee and Lavie, 2005). Other metrics compare the
semantic differences using graphs (e.g., SPICE (Anderson et al., 2016)) or the
probability of word sequences (e.g., perplexity (Jelinek et al., 2005)). In
general, a metric tries to align well with human judgments, although various
human biases and imperfections exist. Many model-free metrics do not correlate
sufficiently with human judgments, as they occasionally award high scores to
low-quality texts (Gatt and Krahmer, 2018). Still, these metrics are widely
used in text generation research as they are acceptable yet cost-efficient
proxies.

BLEU is, with 69 usages, the most employed model-free metric in our review
(Papineni et al., 2002). It measures the word overlap between a candidate text
and one or more references. The score indicates the precision of generated
text based on the overlapping n-grams. A high score in BLEU indicates that the
evaluated text is very similar to the reference in terms of wording. To
mitigate highly precise yet very short texts, a brevity penalty is added.
Notably, BLEU does not account for grammatical errors and only considers the
n-grams of a text for comparison. NIST alters BLEU to also consider the
informativeness of n-grams by valuing more frequent n-grams lower than n-grams
that appear less frequently (Doddington, 2002). Self-BLEU measures the
diversity of generated samples by applying BLEU to pairs of generations (Zhu
et al., 2018). N-gram-based metrics like BLEU can be susceptible to noisy text
(Vaibhav et al., 2019).

ROUGE measures the recall of n-gram overlap between a generated text and a
single reference (Lin, 2004). Notably, 46 papers use ROUGE from our selected
works. It is also known as ROUGE-$N$, where $N$ stands for the length of the
n-grams used for the calculation (e.g., ROUGE-1 for unigrams, ROUGE-2 for
bigrams, etc.). ROUGE-$L$ specifically considers the longest common
subsequence between the generated text and a reference. Compared to the
precision-based metric BLEU, ROUGE is recall-based, which means that including
n-grams is more important than accurately matching the reference. As an
n-gram-based metric, ROUGE is inherently biased towards lexical similarity (Ng
and Abrecht, 2015). As a countermeasure, ROUGE-WE (Ng and Abrecht, 2015) adds
soft lexical matching through word embeddings (making it a hybrid model-based
metric). Here, the individual vectors of constituent tokens are multiplied to
produce the vector for an n-gram.

METEOR is another overlap metric used by 32 studies from our selection
(Banerjee and Lavie, 2005). While BLEU captures precision and ROUGE recall,
METEOR combines both into a single metric. A reordering penalty term punishes
texts with the correct words that appear in the wrong order.

CIDEr is a consensus-based metric that matches a single sentence to a set of
reference sentences (Vedantam et al., 2015). 15 works use this metric. While
originally developed for the image captioning task, many works apply it to
other fields of text generation like summarization (Li and Liang, 2021; Fabbri
et al., 2021; Lu et al., 2022b). It calculates the cosine similarity between
1- to 4-gram co-occurrences and captures precision, recall, grammaticality,
and importance of words by down-weighting common n-grams.

chrF++ is a n-gram-based quality metric that computes n-grams on a character
level instead of the token level (Popović, 2017). This makes chrF++
capabilities independent from language and tokenization. It improves
correlation to human judgments thanks to added uni- and bigrams on the word
level. A total of 13 out of 136 papers use a variant of chrF++.

Distinct-n shows the number of distinct n-grams divided by the total number of
generated tokens (Li et al., 2016). Researchers use this metric to measure the
diversity of a sentence, penalizing sentences with repeated words. It is not
capable of capturing text quality. Most works use $n\in{1,2,3}$ (Yang et al.,
2019), and we find eight works using Distinct-n.

Perplexity in text generation is defined as the degree of uncertainty of a
model in predicting a new sequence (Jelinek et al., 2005). Specifically,
perplexity analyzes the likelihood of word sequences, comparing the generated
probability distribution of words with the actual input. A lower perplexity
score means that the trained model is more capable of generation with varying
input data because it is less perplexed/confused. Perplexity is only a measure
of certainty and, therefore, does not necessarily correlate with the quality
of the generated text. Still, it is a widely used measurement indicating how
confident a model is during generation. 23 papers use perplexity in our
review.

Word Error Rate measures the amount of edits required (substitutions,
deletions, insertions) until a sequence equals the reference. Therefore, the
Word Error Rate measures the similarity of sentences, indicating textual
quality when comparing a result with a reference text. Even though 11 studies
from our assessed works use it, the Word Error Rate is limited in its
capabilities. It does not capture semantic meaning and, therefore, does not
distinguish between contextually relevant or unimportant words. Most mentions
of the Word Error Rate from our selection of works are either reviewing
existing metrics (Fatima et al., 2022) or applying it to speech recognition
(Huang et al., 2018) due to its higher sensitivity to specific error types
(substitutions, deletions, insertions) compared to more flexible n-gram-based
approaches.

SPICE uses a unique approach to measure semantic similarity by implementing a
distance measurement between two texts’ scene graphs that encode objects,
attributes, and relations (Anderson et al., 2016). SPICE ignores fluency, and
a highly similar text pair might not be well-formed, but it captures semantic
similarity more effectively than n-gram-based metrics. This metric is also
used for tasks like image captioning because it can capture relations between
objects and attributes well (Keneshloo et al., 2019). For other tasks, SPICE
can serve as an additional measurement besides other metrics, especially
because it is one of the few model-free metrics that do not rely on n-gram
comparisons (Lin et al., 2020; Lu et al., 2022b).

###  4.3. Model-Based

With model-based metrics, we capture the semantic meaning and potentially
other attributes of a sentence through the use of neural models like BERT
(Devlin et al., 2019). Thus, model-based metrics provide a more flexible
measurement of textual characteristics. This relatively novel category of
metrics generally outperforms many well-established overlap metrics like BLEU
(Papineni et al., 2002; Babakov et al., 2022). Considering LMs’ increasing
capabilities, model-based metrics are employed 28 out of 136 times in the
papers from our reading list, but most studies still include model-free
metrics as a widely accepted rule-based measurement as indicated in Table 2.
Ji et al. (2023) point out that neural models are prone to produce errors in
predicting textual similarity or other attributes. Thus, relying on model-
based metrics alone can propagate errors that adversely affect the models’
accuracy. Moreover, model-based metrics are inherently prone to
insensitivities, biases, and loopholes (He et al., 2023). For example,
BERTScore (Zhang et al., 2020a) is confused by truncation errors in
summarization. We differentiate between fully end-to-end trained metrics that
rely on handcrafted features and/or learned embeddings (e.g., BLEURT (Sellam
et al., 2020), BARTScore (Yuan et al., 2021)) and hybrid metrics that combine
trained elements (e.g., contextual embeddings) with computational logic (e.g.,
BERTScore (Zhang et al., 2020a)). Model-based metrics (trained or hybrid) take
advantage of available rating data for their training and should be more
robust to distribution drifts than model-free metrics (Sellam et al., 2020).

BERTScore is, with 13 occurrences, the most-used model-based metric in our
review (Zhang et al., 2020a). It is a hybrid metric that leverages contextual
token similarity to measure the overlap of a text with a reference. More
specifically, BERTScore uses pairwise cosine similarity to match the word
embeddings between a source and reference text. As it considers precision,
recall, and f1 measure, BERTScore can be applied to various text generation
tasks and correlates well with human judgments. The metric is highly
expressive and inherently tunable for task-specific properties (e.g., fluency,
style, and coherence), although the reliability varies between properties.

MoverScore uses contextualized representations (e.g., BERT embeddings (Devlin
et al., 2019)) to measure the similarity between two texts (Zhao et al.,
2019). It uses the Word Mover Distance (Kusner et al., 2015) to determine how
much one set of embeddings needs to be changed to transform into the reference
embeddings. Thus, MoverScore considers not only the intersection but also the
deviation from the reference. Sentence Mover’s Similarity (Clark et al., 2019)
extends this idea by also including sentence embeddings, improving correlation
with human judgments. Six papers use MoverScore as a similarity metric.

Word Mover Distance calculates the dissimilarity of two texts (Kusner et al.,
2015) using static word representations (e.g., word2vec (Mikolov et al.,
2013)). While Word Mover Distance relies on traditional static word embeddings
that can be visualized for interpretability, some researchers prefer
MoverScore as a measurement due to its greater capability in capturing
context. We find two works from after 2017 that leverage this metric.

BLEURT is a fully learned metric that models human judgment on text quality
(Sellam et al., 2020). BLEURT is highly adaptable, trained end-to-end, and can
be fine-tuned to fit other tasks. However, hybrid metrics like BERTScore
(Zhang et al., 2020a) may show better results with little training data
available and do not rely on the assumption that train and test data are
similarly distributed.

BARTScore leverages a text-to-text model to asses text from perspectives like
informativeness, coherence, and factuality (Yuan et al., 2021). While it can
also capture precision and recall, BARTScore also considers faithfulness and
can be prompted or fine-tuned to fit more specific tasks. BARTScore++ improves
on BARTScore by considering the explicit and implicit error distances when
detecting hallucinations (Lu et al., 2023). The application of BARTScore
heavily depends on the task, requiring a user to think about which variation
to choose and how to design the prompt.

##  5\. Related Challenges

Although text generation has seen much progress concerning its proposed
techniques, related challenges remain that have not yet been solved. Aside
from the specific challenges and limitations in Section 3, we identify nine
main challenges related to text generation. The challenges are bias,
reasoning, factuality, misuse, datasets, interpretability, transparency,
privacy, and computing. This section describes each challenge while surveying
their state-of-the-art mitigation methods and remaining research gaps.

###  5.1. Bias

Bias, in the most general sense, describes systematically erroneous output
that portrays scenarios distortedly (Sheng et al., 2019). In text generation,
this distortion often leans towards or against certain demographic groups or
concepts and amplifies biases in the training sets (Sun et al., 2019). As bias
can occur in different shapes, this problem touches several other areas, such
as toxicity in text generation (Zheng et al., 2023) and sentiment-controlled
text generation (Ghosh et al., 2017; Huang et al., 2020; Kawamae, 2023; Liu et
al., 2021). Bias towards groups or concepts is a pressing problem for machine-
generated text (Sheng et al., 2019; Dhamala et al., 2021). Reducing bias can
reduce the subliminal influence on perceived stereotypes by users of such
systems and provide more neutral answers not steered by ideologies.

While the characteristics of bias are manifold, we identify three main types:
exposure bias, allocation bias, and representation bias. First, exposure bias
occurs when a model is only exposed to the training data distribution instead
of its prediction (Keneshloo et al., 2019). To avoid exposure bias, Keneshloo
et al. (2019) remove the ground-truth dependency during training and use only
the model distribution to minimize the loss function. Reinforcement learning
concerning a metric like ROUGE (Lin, 2004) can mitigate exposure bias (Wang et
al., 2018; Jin and Gildea, 2022). A Generative Adversarial Network can be
employed as the generator constantly relies on its output during training
(Shen et al., 2022a). Second, allocation bias appears when models perform
better on data associated with groups or situations overrepresented in the
training data (Sun et al., 2019). Zhao et al. (2018) show how gender-specific
allocation bias can be mitigated by creating a second dataset with a swapped
gender though it doubles the dataset size and is expensive to create. Third,
representation bias appears when associations between groups with certain
concepts are captured in word embeddings or model parameters. Thus, the
representation (e.g., the embedding) reflects the bias (Sun et al., 2019). An
intuitive way to tackle representation bias is to debias the embeddings
directly by regularization (Huang et al., 2020). This comes with a trade-off
between generation quality and control because it can impose the target
attribute on the LM on improper positions (Gu et al., 2022; Huang et al.,
2020). Debiasing can be effective while only fine-tuning 1% of parameters on
an unbiased dataset, making learned techniques relatively cheap to adopt (Gira
et al., 2022). Notably, Schick et al. (2021) show that pre-trained LMs are
capable of recognizing their own biases to a considerable degree. Thus, they
can be prompted to self-diagnose their output.

The quantification of bias is especially relevant when considering an LM for
real-world applications and determining the effectiveness of mitigation
techniques. BOLD (Dhamala et al., 2021) and FairPrim (Fleisig et al., 2023)
both are helpful datasets to measure various biases (profession, gender, race,
religious belief, and political ideology). BOLD metrics (Dhamala et al., 2021)
further assess the texts’ sentiment, toxicity, regard, psycholinguistic norms,
and gender polarity. Notably, not all biases are easy to measure with current
metrics due to either a lack of labeled data specific to the bias type or
corresponding noise in the supposedly bias-free data.

###  5.2. Reasoning

Reasoning describes the capability of a text generation system to infer its
choices and writing from the source context logically and sensibly. A model
should be able to make an understandable decision and explain its reasoning to
a human. Several works point out that, even though recent years yielded
significant improvements, LMs have difficulties with such commonsense
inference (Lin et al., 2020; Zellers et al., 2019a). Thus, eliciting reasoning
with text generation is an active research topic. The lack of reasoning
capabilities comes from misunderstanding facts in the source context and is
also referred to as intrinsic hallucinations (Ji et al., 2023) (more details
in Section 5.3). Text generation systems require reasoning capabilities for
complex tasks like story generation (Liang et al., 2023), question answering
(Fan et al., 2019), or math problems (Cobbe et al., 2021).

One popular approach to improve reasoning is Chain-of-Thought Prompting (CoT)
(Wei et al., 2022). Here, the model is prompted to think step by step and
output its thought process in natural language. It improves performance for
arithmetic, commonsense, and symbolic reasoning tasks. Exchange-of-Thought
maintains the reasoning capabilities of CoT (Wei et al., 2022) while allowing
for an iterative self-interaction that improves the output text (Yin et al.,
2023). The approach imitates a multi-turn conversation between experts on a
task. Chen et al. (2020) survey another method to improve reasoning
capabilities via path-finding strategies on a knowledge graph. We also find
considerable research related to testing a model’s reasoning capabilities.
Stolfo et al. (2023) explain how to specifically test the robustness of
mathematical reasoning by grounding a behavioral analysis in a causal graph.
The CommonGen dataset (Lin et al., 2020) requires relational reasoning and
compositional generalization. The HellaSwag dataset (Zellers et al., 2019a)
can also be used to test a model’s ability of generative commonsense
reasoning.

###  5.3. Hallucinations

Hallucinations describe outputs that are factually wrong or unfaithful to the
source. We distinguish between intrinsic and extrinsic hallucinations, which
refers to what content the generated text contradicts to (Ji et al., 2023).
Intrinsic hallucinations contradict the source content, so they are unfaithful
to the training data or input text. Extrinsic hallucinations describe outputs
that can neither be supported nor contradicted by the source content (Maynez
et al., 2020).

The main reasons for hallucinations emerge from the source data because
diversity-valuing tasks like dialogue generation (Ji et al., 2023) inherently
generate text that deviates from factual information in the source data. Bias
in the training data and text generation models can also lead to factually
wrong outputs (Qader et al., 2018; Ji et al., 2023). Improving factuality in
text generation makes systems more reliable in real-world applications (e.g.,
chat support assistants and summarization systems).

Overall, models pre-trained on large amounts of data tend to hallucinate less
(Maynez et al., 2020; Zhou et al., 2021). Prabhumoye et al. (2021) mitigate
hallucinations in document-grounded text generation by implementing a
selective attention variant for the documents’ contextualized representations.
Faithfulness to a source can be improved by representing the knowledge in a
simpler relational format and verifying it against relation tuples from the
source or reference (Goodrich et al., 2019).

Besides mitigating bias, other works employ mechanisms to detect
hallucinations in existing text. Measuring hallucinations with LMs can be
challenging they are designed to predict texts of high linguistic quality
(e.g., fluency, grammaticality) rather than factual content (Gatt and Krahmer,
2018). In addition, most automatic metrics do not correlate with faithfulness
or factuality, so they are often unreliable, highlighting human evaluation on
factuality datasets (Abhishek et al., 2022; Zhou et al., 2021) as the
preferred method (Maynez et al., 2020; Goyal and Durrett, 2020; Falke et al.,
2019; Ji et al., 2023). Goyal and Durrett (2020) present a textual entailment
system that can fact-check generated text and localize the factual error. The
model-based metric BARTScore++ can detect hallucinations to a certain degree
because it imitates human-like error analysis (Lu et al., 2023). Fabbri et al.
(2022) propose QAFactEval to evaluate factual consistency for the QA task.
Madaan et al. (2021) propose a model to automatically generate counterfactual
text that can be leveraged to create a synthetic test set on factuality.

###  5.4. Misuse

It is important to name the adversarial risks of text-generative models. Only
when knowing how these systems can be misused countermeasures can be
researched. This is also referred to as threat modeling (Zellers et al.,
2019b), i.e., identifying threats and vulnerabilities from an adversary’s
point of view. We identify three main categories of how text generation
systems are misused currently or in the near future. These are non-disclosed,
misaligned, and adversarial usage.

Machine-generated systems can pose a threat if their usage is not disclosed
properly. Convincing machine-generated text poses a significant challenge to
academic integrity (Cotton et al., 2024). Additionally, highly controllable
text generation systems can flood social media (Han et al., 2023; Zellers et
al., 2019b; Tourille et al., 2022) or product pages (Adelani et al., 2020)
with intentionally biased and non-factual information. The main mitigation
methods for undisclosed usage concentrate on discriminating machine-generated
text from human-written text (Tourille et al., 2022; Wahle et al., 2022b,
2021; Fagni et al., 2021; Munir et al., 2021) although attribution alone does
not verify a text’s factuality (Schuster et al., 2020). Current classifiers
for machine-generated text suffer from regular false positive detections for
many reasons ranging from shortness to incoherence (Jawahar et al., 2020). An
adversary using multiple text generators within the same document might pose
an additional challenge for detectors (Uchendu et al., 2021). Rodriguez et al.
(2022) suggest that paragraph-level detectors can be used to detect the
tampering of full-length documents. GLTR utilizes the fact that human writing
includes more sequentially improbable tokens in a sequence than machine text
and highlights tokens probabilities (Gehrmann et al., 2019). Zhong et al.
(2020) specifically assess the factual structure of a text to detect machine
samples.

Users of text generation systems might have harmful intentions and
specifically tailor prompts that lead to text generation that is not aligned
with human ethics or safety regulations. To mitigate misaligned use, many
models like GPT-4 (Achiam et al., 2023), LLaMA 2 (Touvron et al., 2023), or
Gemini (Team et al., 2023) include automated censorship mechanisms that block
potentially harmful content. For example, Gemini (Team et al., 2023) leverages
advanced Chain-of-Thought recipes to align with safety policies. GPT-4 (Achiam
et al., 2023) provides an additional reward signal during RLHF fine-tuning
(Ouyang et al., 2022) that targets correct behavior, such as refusing to give
harmful responses. However, these mechanisms can be tricked. For example,
Jiang et al. (2024) leverage the poor performance of many modern LMs in
recognizing ASCII art to jailbreak safety-aligned LMs.

A more technical misuse of machine-generated text is adversarial. As text
classification systems are sensitive to certain words, an adversary could
intentionally generate text including certain words, ultimately hindering the
effectiveness of classification models (Mosca et al., 2022; Gao et al., 2018;
Jin et al., 2020). Due to the high instruction-following capabilities of
modern LMs (Ouyang et al., 2022), an additional risk arises from prompt
injections. Prompt injections describe secretly hidden text snippets (prompts)
in any content used by an LM-based application (Greshake et al., 2023). They
can lead to the extraction of sensitive information from LMs (e.g., personal
data, credentials) (Greshake et al., 2023). Prompt injections can also be used
for phishing, scams, or masquerading, and some systems might be vulnerable to
unwanted API calls. To avoid tailored texts tricking classifiers, Mosca et al.
(2022) propose a logits-based metric that captures words with a suspiciously
high impact on a classifier. The risks of text-generative models are abundant,
and mitigation methods must be tailored to each problem. Meanwhile, we
emphasize raising awareness of the diverse problems to foster the development
of mitigation methods.

###  5.5. Datasets

The data scale has increased significantly since contemporary LMs need to be
pre-trained on large amounts of web-scraped data. As corpora grow, finding and
filtering open-source and high-quality data becomes more challenging, which
might encourage exploring licensed data for training (e.g., transcribed
YouTube videos). Also, many datasets and their sources are unstandardized
(Fatima et al., 2022).

Fleisig et al. (2023) make several suggestions on how to improve the quality
of datasets. Datasets should clarify the potential risks (e.g., toxicity
(Gehman et al., 2020)) of the provided machine-generated text. With high-
quality annotations, authors can measure or mitigate a diverse set of
fairness-related harms, and identify the demographic groups of annotators
(Fleisig et al., 2023). In addition, datasets should account for annotator
disagreement, providing individual annotators’ judgments rather than a final
score. Context-dependent harms can only be identified when the context is
provided in the dataset (e.g., perceived author of text, application), and a
diverse set of annotators can ensure that various demographic groups are
represented.

A key problem related to data scarcity concerns multilingual applications.
Text generation works best for languages encountered frequently during pre-
training. However, most text generation research takes place in the United
States, followed by highly developed countries such as China, the United
Kingdom, and India (Ramos et al., 2023; Abdalla et al., 2023). Thus, most
research is performed on corresponding corpora, mostly in English or Chinese.
Low-resource languages remain underexplored in machine-generated text, and
multi-lingual models must be specifically trained to fit the niche (Shaham et
al., 2024).

To enhance a model’s performance on scarce data like low-resource languages,
researchers apply variations to the training process. Shaham et al. (2024)
show that just 40 multilingual samples in an English tuning set improve multi-
lingual instruction following. This also generalizes to unseen languages
during fine-tuning, indicating that including two to four languages in the
training set significantly improves cross-lingual generalization. Datasets
should not be used repeatedly as a benchmark, as overfitting reduces its
efficiency long term (Fleisig et al., 2023). Thus, there exists a constant
need for new datasets. One example dataset is XWikiRef (Taunk et al., 2023), a
Wikipedia summarization dataset on eight low-resource languages across five
domains (i.e., books, films, politicians, sportsmen, and writers).

###  5.6. Interpretability

A text generation model is interpretable when the user can understand why a
certain output was produced. Modern LMs inherently do not provide intuitive
insight into how textual outputs are produced, as their weights and
representations appear mathematically complicated (Lu et al., 2022a).

Interpretability is a desired goal when solving tasks because, without the
reasoning behind the final output being clear, the correctness of the output
can not be easily assessed, and users need to rely solely on trusting the
model. Moreover, researchers rely on the intricate analysis of existing models
to propose novel ideas.

Singh et al. (2024) point out that, overall, two aspects of interpretation
research need to be explained. First, we require an intricate explanation of
an LM’s behavior. This comprises local characteristics like feature
attribution and data grounding and global characteristics like attention head
importance and data influence. Second, a clear explanation of used datasets is
required. This includes interactive clarification using natural language and
aiding data analysis. Tenney et al. (2020) propose a tool to visualize an LMs’
behavior on the architectural level. They visualize the sentence embeddings,
classification probabilities, attention, and confusion matrix. By providing
the information in a visual format that is easy to understand, one can
directly assess how an LM behaves when specific data points are altered.
Ultimately, tools like this provide a meaningful step towards
interpretability, as models’ inner processes usually remain hidden and their
disclosure requires time-consuming development of custom solutions.

###  5.7. Transparency

Transparency involves two major aspects. First, transparency refers to the
lucid explanation and documentation of proposed models and datasets. This
includes the research and corporative proposals along with their underlying
methodologies, which many newly released models like GPT-4 (Achiam et al.,
2023) and Gemini (Team et al., 2023) only provide superficially. The other
aspect of transparency with text generation is the disclose of its usage in
academic writing. Since big tech companies increasingly lead and,
consequently, shift NLP research directions, the discussion of potential risks
(e.g., toxicity, factuality) has become essential in the last years,
especially when releasing LMs (Abdalla et al., 2023). To research and address
the emerging challenges, the transparent reporting of model characteristics is
of key importance.

For this, Mitchell et al. (2018) provide an extensive and standardized model
card that can be shipped together with novel LMs. However, corporate
organizations often have a monetary interest in releasing new models, and
fully disclosing all model details can conflict with a company’s interest in
not having their product replicated or released to an open-source audience.
Aczel and Wagenmakers (2023) describe the transparent and credible usage of
text-generative models in scientific writing as threefold. First, researchers
should disclose the model and its training data used. In addition, the
generated text, or a summary of it, should be available to the readers.
Second, the authors should specify for which aspects of a scientific work
(outline, writing, code) a model was used. Third, researchers are required to
always verify generated text which includes a check on plagiarism. Fulfilling
all aspects described by Aczel and Wagenmakers (2023) can be difficult, but
the guidelines give an idea of the ideal process for documenting AI usage.
Wahle et al. (2023c) propose AI Usage Cards, a shorter card that can be
included in research papers to document how text generation models are used
for the work.

###  5.8. Privacy

The data collection process raises privacy concerns as there is apprehension
about companies using human inputs to their system to continuously train their
models. Modern LMs require large amounts of data crawled from the web for
their pre-training (Touvron et al., 2023) and it is hard to control whether
personal data is being used. These models have been found to output memorized
passages from their training data (Yuan et al., 2022; Carlini et al., 2020).
Thus, LMs could also leak private information they encountered during training
(Li et al., 2021; Greshake et al., 2023).

Li et al. (2023a) show how multi-step jailbreaking prompts can be formulated
to expose private information like names or emails, even bypassing safety
modules employed in applications like ChatGPT (Yuan et al., 2022). Examples
like this show that privacy-related research about text generation is crucial.

Yue et al. (2023) use differential privacy to mathematically guarantee that
private information can neither be generated nor reconstructed from the
output. The key idea of differential privacy is that including or excluding a
single individual’s data should not significantly affect the model’s output.
By understanding the sensitivity regarding single data points, the algorithm
knows how much noise to add to a stochastic gradient descent algorithm. Once
the noise has been added, further data processing will not diminish privacy
protection. This property is crucial because it ensures privacy is not
compromised through additional analysis. However, their mathematical guarantee
of privacy protection comes with a trade-off regarding output quality.
Notably, tight differential privacy harms small classes like minority groups
during training as rarely mentioned groups could be nullified in the process.
Thus, one needs to find the right balance between privacy and quality in real-
world applications. Song and Shmatikov (2019) develop an auditing model that
lets users check whether their data has been used to train a text generation
model.

###  5.9. Computing

LMs have been growing increasingly large (Zhao et al., 2023) and with more
data and a higher number of model parameters, training times also surge. This
requires more computing and energy to publish a novel state-of-the-art model.
Nowadays, this becomes a problem for small companies and research
organizations as only big tech companies like OpenAI, Google, or Meta have the
resources to train a competitive language model from the ground up (Abdalla et
al., 2023). Larger models also have a higher environmental cost, emitting more
carbon during training than smaller models (Touvron et al., 2023; Fabbri et
al., 2022). Reducing the computational requirements of recent LMs could
potentially lower their environmental impact due to lower power consumption.

The scaling laws from (Kaplan et al., 2020) suggest that larger models will
continue to perform better than smaller ones while reaching the same level of
performance with fewer optimization steps. They also find that a model’s
performance depends strongly on the number of parameters, dataset size, and
the amount of computing, while architectural parameters like layer depth or
width are not as important. Thus, making text generation models lightweight
and improving computing times is an active research topic. Liu et al. (2023a)
propose a transformer with ternary (i.e., -1, 0, 1) or binary (i.e., 0, 1)
weights which facilitate multiplication-free computations. Their variation of
BART (Lewis et al., 2020) is up to 16 times more efficient than its real-
valued counterpart while achieving close to normal performance. However,
binarization and ternarization require bit-packing to have actual memory
savings and need dedicated hardware support for real-time acceleration. Li and
Liang (2021) propose prefix-tuning, a light-weight alternative to fine-tuning
where only 0.1% of a models parameters are optimized. Their model performs
comparable to a full-data setting and is even better in low-data settings and
extrapolation on unseen topics. Approaches to reduce LMs’ environmental impact
are often not included in the works we read. We identify that NLP-focused
research on reducing environmental harm is scarce but necessary if the field
wants sustainable future research.

##  6\. Epilogue

###  6.1. Conclusion

In this systematic literature review, we covered 244 publications focusing on
recent developments in text generation since the Transformer architecture and
large language models in 2017. In that context, we identified and discussed
the main tasks and sub-tasks in the field, assessed how text generation
systems are evaluated, and outlined relevant challenges that researchers can
attack in the near and mid-term future. To summarize this work, we revisit our
initial research questions.
What constitutes the task of text generation? What are the main sub-tasks?
We categorized text generation into five main sub-tasks: open-ended text
generation, summarization, translation, paraphrasing, and question answering.
We identified prominent sub-tasks and unique challenges to display recent
research within each of these. Open-ended text generation comprises open-
domain applications, story generation, and dialogue generation. Open-ended
tasks are inherently challenged by trying to consistently model long contexts,
especially when considering sub-tasks that are lexically and semantically
diverse, like dialogue generation or story generation. In summarization, we
focussed on single documents, multiple documents, and dialogues. Recent
studies concentrate on sub-tasks like multi-document summarization and
dialogue summarization, which exhibit higher contextual complexity. While
sentence-level translation is a well-established task, document-level
translation poses a chance to mitigate limitations like word ambiguities
within or between documents. Paraphrasing research shifts from uncontrolled
generation towards more controllable scenarios. Here, we noticed a lack of
aligned paraphrasing datasets that provide labels for such controllable
attributes. Question answering relies on internal and external knowledge
sources and is challenged by reasoning, non-answerability, and non-factoid
questions.
How are text generation systems evaluated? What are the concomitant
limitations?
We identified two main methodologies for evaluation: model-free and model-
based. While model-free evaluations are based on static and rule-based
algorithms, model-based ones leverage word or sentence embeddings of pre-
trained models to produce a score. Most model-free metrics rely on n-gram
overlap, but their correlation to quality as humans perceive it varies.
Moreover, some textual characteristics, like factuality or bias, are difficult
to measure using model-free metrics, which is why we emphasized the need for
additional model-based metrics and human evaluation. In addition to quality,
model-based metrics can capture attributes like factuality or writing style by
fine-tuning or prompting. However, as neural models are subject to errors,
model-based metrics might come with various biases and not reliably quantify
these attributes, while additionally lacking interpretability.
What are the open challenges in text generation?
We covered bias, reasoning, hallucinations, misuse, datasets,
interpretability, transparency, privacy, and computing. In this work, we
identified three prominent bias types relevant to text generation: exposure-,
allocation-, and representation bias. Recent works are concerned with bias
measurement and mitigation in generated texts. External modules that quantify
and detect hallucinations can provide transparency and improve factuality in
text generation. Specific prompting techniques like Chain-of-Thought stimulate
language models’ reasoning capabilities, but they require an intricate prompt
design. We outlined various scenarios of how text generation could potentially
be misused. These scenarios are categorized into non-disclosed, misaligned,
and adversarial usage. Additionally, existing datasets for text generation
tend to disregard low-resource languages and are sometimes not standardized
into a uniform format or lack metadata like inter-annotator agreement scores.
Interpretability in text generation should consider both the used models and
associated datasets. Differential privacy can mitigate the risk of leaking
private information from training data at the cost of output quality. To make
capable language models more accessible to the community, we find incipient
research about how to make the compute requirements of text generation systems
lightweight without sacrificing substantial amounts of performance. This
involves both the training process and the inference of models.
What are prominent research directions in text generation?
Recent publications have focused more on improving the intrinsic (e.g., bias,
factuality) and contextual (e.g., relevance, completeness) quality of the
machine-generated text. We suspect research in these areas will be
particularly fruitful, and we highlighted various directions accordingly. For
example, we identified that many bias mitigation methods depend on careful
data selection and augmentation while only working for specific bias types.
Research towards more universal mitigation techniques, possibly leveraging
self-aware language models to quantify their own biases, would provide an
impactful contribution to the community. Improvements in factuality are
challenged by ponderous evaluation. To foster additional research in that
area, we suggested investigating the effective quantification of
hallucinations using model-based methods. Aside from prompting strategies,
enhancing language models’ reasoning capabilities could be considered during a
model’s training instead of during inference. With language model applications
permeating people’s everyday lives, subordinate problems like its misuse
become apparent. To ensure the safety and responsible use of these text-
generative models, safety-aligned systems should be able to detect when they
are about to be exploited by misaligned or adversarial prompts. Excluding
private information from both the training process of language models and the
generation of text poses another opportunity for research. Compared to more
established tasks like uncontrolled paraphrasing or single-document
summarization, contemporary proposals suggest that the community’s general
interest shifted towards more specific scenarios (e.g., controlled
paraphrasing) and sub-tasks that are contextually more complex (e.g., dialogue
summarization). We suggest that such scenarios will be more prominently
researched in subsequent works. Additionally, we identified the potential to
use large pre-trained models that can operate across a variety of domains and
tasks. Future work could focus on more effectively leveraging the internal
knowledge of such models for tasks like question answering while avoiding
hallucinated content. In summary, we discussed several sub-tasks of text
generation that leave space for future research and outlined key challenges
that can be targeted in the near and mid-term future.

###  6.2. Limitations

Text generation can involve various other modalities that are not immediately
natural language. For example, methods include table generation (Lu et al.,
2022b), table-to-text generation (Wang et al., 2020), sql-to-text generation
(Xu et al., 2018), or text generation from an abstract meaning representation
(Mager et al., 2020). In this work, we focus on text generation through
natural language input mainly because it is the most prominent approach in
current NLP research and it generates text that can be unified across many
tasks (e.g., summarization, translation). Some recently published works may
not have passed the filters of our systematic review because they might not
have had enough time to accumulate enough citations. Thus, we perform a manual
assessment that includes additional works to our reading list, but of course,
works that could have been relevant might also be left out of that assessment.
We tried to mitigate this by constantly conversing about the covered topics
and judging their relevance by voting between the authors. We release these
voting judgments to the public through our GitHub repository1. When retrieving
papers, we rely on Semantic Scholars’ set of indexed papers, which might lead
to some papers being left out, although Semantic Scholar’s coverage,
particularly in AI and NLP, is high.

###### Acknowledgements.

This work was supported by the Lower Saxony Ministry of Science and Culture
and the VW Foundation.

## References

  * (1)
  * Abdalla et al. (2023) Mohamed Abdalla, Jan Philip Wahle, Terry Ruas, et al. 2023\.  The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 13141–13160.  https://doi.org/10.18653/v1/2023.acl-long.734
  * Abhishek et al. (2022) Tushar Abhishek, Shivprasad Sagare, Bhavyajeet Singh, et al. 2022\.  XAlign: Cross-lingual Fact-to-Text Alignment and Generation for Low-Resource Languages. In _Companion Proceedings of the Web Conference 2022_. ACM.  https://doi.org/10.1145/3487553.3524265
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2023\.  GPT-4 Technical Report.
  * Aczel and Wagenmakers (2023) Balazs Aczel and Eric-Jan Wagenmakers. 2023.  Transparency Guidance for ChatGPT Usage in Scientific Writing.  https://doi.org/10.31234/osf.io/b58ex
  * Adelani et al. (2020) David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, et al. 2020\.  Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection.  In _Advanced Information Networking and Applications_. Springer International Publishing, 1341–1354.  https://doi.org/10.1007/978-3-030-44041-1_114
  * Alabdulkarim et al. (2021) Amal Alabdulkarim, Siyan Li, and Xiangyu Peng. 2021.  Automatic Story Generation: Challenges and Attempts. In _Proceedings of the Third Workshop on Narrative Understanding_. Association for Computational Linguistics, Virtual, 72–83.  https://doi.org/10.18653/v1/2021.nuse-1.8
  * Alihosseini et al. (2019) Danial Alihosseini, Ehsan Montahaei, and Mahdieh Soleymani Baghshah. 2019\.  Jointly Measuring Diversity and Quality in Text Generation Models. In _Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation_. Association for Computational Linguistics, Minneapolis, Minnesota, 90–98.  https://doi.org/10.18653/v1/W19-2311
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: Semantic Propositional Image Caption Evaluation.
  * Artetxe et al. (2023) Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan, and Luke Zettlemoyer. 2023.  Revisiting Machine Translation for Cross-lingual Classification.
  * Babakov et al. (2022) Nikolay Babakov, David Dale, Varvara Logacheva, and Alexander Panchenko. 2022.  A large-scale computational study of content preservation measures for text style transfer and paraphrase generation. In _Proc. of ACL_. Association for Computational Linguistics, Dublin, Ireland, 300–321.  https://doi.org/10.18653/v1/2022.acl-srw.23
  * Bandel et al. (2022) Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer, et al. 2022\.  Quality Controlled Paraphrase Generation. In _Proc. of ACL_. Association for Computational Linguistics, Dublin, Ireland, 596–609.  https://doi.org/10.18653/v1/2022.acl-long.45
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_. Association for Computational Linguistics, Ann Arbor, Michigan, 65–72.
  * Bao et al. (2020) Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020.  PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable. In _Proc. of ACL_. Association for Computational Linguistics, Online, 85–96.  https://doi.org/10.18653/v1/2020.acl-main.9
  * Becker et al. (2023) Jonas Becker, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2023\.  Paraphrase Detection: Human vs. Machine Content.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020\.  Longformer: The Long-Document Transformer.
  * Bengio et al. (2000) Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000.  A Neural Probabilistic Language Model. In _Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA_ , Todd K. Leen, Thomas G. Dietterich, and Volker Tresp (Eds.). MIT Press, 932–938.
  * Botha et al. (2018) Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. 2018.  Learning To Split and Rephrase From Wikipedia Edit History. In _Proc. of EMNLP_. Association for Computational Linguistics, Brussels, Belgium, 732–737.  https://doi.org/10.18653/v1/D18-1080
  * Bražinskas et al. (2020) Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020\.  Few-Shot Learning for Opinion Summarization. In _Proc. of EMNLP_. Association for Computational Linguistics, Online, 4119–4135.  https://doi.org/10.18653/v1/2020.emnlp-main.337
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, et al. 2020\.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).
  * Cai et al. (2020) Lin-Qin Cai, Min Wei, Si-Tong Zhou, and Xun Yan. 2020\.  Intelligent Question Answering in Restricted Domains Using Deep Learning and Question Pair Matching.  _IEEE Access_ 8 (2020), 32922–32934.  https://doi.org/10.1109/ACCESS.2020.2973728
  * Callison-Burch et al. (2008) Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008\.  Further Meta-Evaluation of Machine Translation. In _Proceedings of the Third Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Columbus, Ohio, 70–106.
  * Carlini et al. (2020) Nicholas Carlini, Florian Tramer, Eric Wallace, et al. 2020\.  Extracting Training Data from Large Language Models.
  * Chen et al. (2019) Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019\.  Controllable Paraphrase Generation with a Syntactic Exemplar. In _Proc. of ACL_. Association for Computational Linguistics, Florence, Italy, 5972–5984.  https://doi.org/10.18653/v1/P19-1599
  * Chen et al. (2020) Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020.  A review: Knowledge reasoning over knowledge graph.  _Expert Systems with Applications_ 141 (2020), 112948.  https://doi.org/10.1016/j.eswa.2019.112948
  * Chung et al. (2023) John Chung, Ece Kamar, and Saleema Amershi. 2023.  Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2023.acl-long.34
  * Clark et al. (2021) Elizabeth Clark, Tal August, Sofia Serrano, et al. 2021\.  All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text. In _Proc. of ACL_. Association for Computational Linguistics, Online, 7282–7296.  https://doi.org/10.18653/v1/2021.acl-long.565
  * Clark et al. (2019) Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019\.  Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts. In _Proc. of ACL_. Association for Computational Linguistics, Florence, Italy, 2748–2760.  https://doi.org/10.18653/v1/P19-1264
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. 2021\.  Training Verifiers to Solve Math Word Problems.
  * Costa et al. (2018) Felipe Costa, Sixun Ouyang, Peter Dolog, and Aonghus Lawlor. 2018.  Automatic Generation of Natural Language Explanations. In _Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion_. ACM.  https://doi.org/10.1145/3180308.3180366
  * Cotton et al. (2024) Debby Cotton, Peter Cotton, and J. Reuben Shipway. 2024\.  Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT.  (2024).  https://doi.org/10.35542/osf.io/mrz8h
  * Currey et al. (2020) Anna Currey, Prashant Mathur, and Georgiana Dinu. 2020\.  Distilling Multiple Domains for Neural Machine Translation. In _Proc. of EMNLP_. Association for Computational Linguistics, Online, 4500–4511.  https://doi.org/10.18653/v1/2020.emnlp-main.364
  * Deng et al. (2021) Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021.  Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation. In _Proc. of EMNLP_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7580–7605.  https://doi.org/10.18653/v1/2021.emnlp-main.599
  * Dennett (2013) Daniel C Dennett. 2013\.  The role of language in intelligence.  _Sprache und Denken/Language and Thought_ (2013), 42.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proc. of NAACL-HLT_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, et al. 2021\.  BOLD. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. ACM.  https://doi.org/10.1145/3442188.3445924
  * Doddington (2002) George Doddington. 2002\.  Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In _Proceedings of the second international conference on Human Language Technology Research -_. Association for Computational Linguistics, San Diego, California, 138.  https://doi.org/10.3115/1289189.1289273
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically Constructing a Corpus of Sentential Paraphrases. In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Dou et al. (2022) Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2022.  Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text. In _Proc. of ACL_. Association for Computational Linguistics, Dublin, Ireland, 7250–7274.  https://doi.org/10.18653/v1/2022.acl-long.501
  * Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017.  Learning to Ask: Neural Question Generation for Reading Comprehension. In _Proc. of ACL_. Association for Computational Linguistics, Vancouver, Canada, 1342–1352.  https://doi.org/10.18653/v1/P17-1123
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. In _Proc. of ACL_. Association for Computational Linguistics, Online, 5055–5070.  https://doi.org/10.18653/v1/2020.acl-main.454
  * Egonmwan and Chali (2019) Elozino Egonmwan and Yllias Chali. 2019.  Transformer and seq2seq model for Paraphrase Generation. In _Proceedings of the 3rd Workshop on Neural Generation and Translation_. Association for Computational Linguistics, Hong Kong, 249–255.  https://doi.org/10.18653/v1/D19-5627
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model. In _Proc. of ACL_. Association for Computational Linguistics, Florence, Italy, 1074–1084.  https://doi.org/10.18653/v1/P19-1102
  * Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022.  QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 2587–2601.  https://doi.org/10.18653/v1/2022.naacl-main.187
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, et al. 2021\.  SummEval: Re-evaluating Summarization Evaluation.  _Transactions of the Association for Computational Linguistics_ 9 (2021), 391–409.  https://doi.org/10.1162/tacl_a_00373
  * Fagni et al. (2021) Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021\.  TweepFake: About Detecting Deepfake Tweets.  _PLOS ONE_ 16, 5 (2021), e0251415.  https://doi.org/10.1371/journal.pone.0251415
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019\.  Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference. In _Proc. of ACL_. Association for Computational Linguistics, Florence, Italy, 2214–2220.  https://doi.org/10.18653/v1/P19-1213
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, et al. 2019\.  ELI5: Long Form Question Answering. In _Proc. of ACL_. Association for Computational Linguistics, Florence, Italy, 3558–3567.  https://doi.org/10.18653/v1/P19-1346
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical Neural Story Generation. In _Proc. of ACL_. Association for Computational Linguistics, Melbourne, Australia, 889–898.  https://doi.org/10.18653/v1/P18-1082
  * Fatima et al. (2022) Noureen Fatima, Ali Shariq Imran, Zenun Kastrati, Sher Muhammad Daudpota, and Abdullah Soomro. 2022\.  A Systematic Literature Review on Text Generation Using Deep Neural Network Models.  _IEEE Access_ 10 (2022), 53490–53503.  https://doi.org/10.1109/access.2022.3174108
  * Feng et al. (2021) Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021.  Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization. In _Proc. of ACL_. Association for Computational Linguistics, Online, 1479–1491.  https://doi.org/10.18653/v1/2021.acl-long.117
  * Fleisig et al. (2023) Eve Fleisig, Aubrie Amstutz, Chad Atalla, et al. 2023\.  FairPrism: Evaluating Fairness-Related Harms in Text Generation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2023.acl-long.343
  * Gambhir and Gupta (2017) Mahak Gambhir and Vishal Gupta. 2017.  Recent automatic text summarization techniques: a survey.  _Artificial Intelligence Review_ 47, 1 (2017), 1–66.  https://doi.org/10.1007/s10462-016-9475-9
  * Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018\.  Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers. In _2018 IEEE Security and Privacy Workshops (SPW)_. IEEE.  https://doi.org/10.1109/spw.2018.00016
  * Gao et al. (2019) Yifan Gao, Lidong Bing, Wang Chen, Michael R. Lyu, and Irwin King. 2019.  Difficulty Controllable Generation of Reading Comprehension Questions. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , Sarit Kraus (Ed.). ijcai.org, 4968–4974.  https://doi.org/10.24963/ijcai.2019/690
  * Garg et al. (2021) Sonal Garg, Sumanth Prabhu, Hemant Misra, and G. Srinivasaraghavan. 2021.  Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the State of the Art in Natural Language Generation: Core Tasks, Applications and Evaluation.  _Journal of Artificial Intelligence Research_ 61 (2018), 65–170.  https://doi.org/10.1613/jair.5477
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020.  RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 3356–3369.  https://doi.org/10.18653/v1/2020.findings-emnlp.301
  * Gehrmann et al. (2019) Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019\.  GLTR: Statistical Detection and Visualization of Generated Text. In _Proc. of ACL_. Association for Computational Linguistics, Florence, Italy, 111–116.  https://doi.org/10.18653/v1/P19-3019
  * Ghosh et al. (2017) Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017\.  Affect-LM: A Neural Language Model for Customizable Affective Text Generation. In _Proc. of ACL_. Association for Computational Linguistics, Vancouver, Canada, 634–642.  https://doi.org/10.18653/v1/P17-1059
  * Gira et al. (2022) Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022\.  Debiasing Pre-Trained Language Models via Efficient Fine-Tuning. In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_. Association for Computational Linguistics, Dublin, Ireland, 59–69.  https://doi.org/10.18653/v1/2022.ltedi-1.8
  * Goldberg et al. (1994) E. Goldberg, N. Driedger, and R.I. Kittredge. 1994.  Using natural-language processing to produce weather forecasts.  _IEEE Expert_ 9, 2 (1994), 45–53.  https://doi.org/10.1109/64.294135
  * Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019\.  Assessing The Factual Accuracy of Generated Text. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (Eds.). ACM, 166–175.  https://doi.org/10.1145/3292500.3330955
  * Goyal et al. (2023) Rupali Goyal, Parteek Kumar, and V. P. Singh. 2023\.  A Systematic survey on automated text generation tools and techniques: application, evaluation, and challenges.  _Multimedia Tools and Applications_ (2023).  https://doi.org/10.1007/s11042-023-15224-0
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating Factuality in Generation with Dependency-level Entailment. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 3592–3603.  https://doi.org/10.18653/v1/2020.findings-emnlp.322
  * Greshake et al. (2023) Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, et al. 2023\.  Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In _Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security_. ACM, Copenhagen Denmark, 79–90.  https://doi.org/10.1145/3605764.3623985
  * Grusky (2023) Max Grusky. 2023\.  Rogue Scores. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2023.acl-long.107
  * Gu et al. (2022) Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, et al. 2022\.  Improving Controllable Text Generation with Position-Aware Weighted Decoding. In _Findings of the Association for Computational Linguistics: ACL 2022_. Association for Computational Linguistics, Dublin, Ireland, 3449–3467.  https://doi.org/10.18653/v1/2022.findings-acl.272
  * Guan et al. (2021) Jian Guan, Xiaoxi Mao, Changjie Fan, et al. 2021\.  Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence. In _Proc. of ACL_. Association for Computational Linguistics, Online, 6379–6393.  https://doi.org/10.18653/v1/2021.acl-long.499
  * H. Lee et al. (2020) Helena H. Lee, Ke Shu, Palakorn Achananuparp, et al. 2020\.  RecipeGPT: Generative Pre-training Based Cooking Recipe Generation and Evaluation System. In _Companion Proceedings of the Web Conference 2020_ _(WWW ’20)_. Association for Computing Machinery, New York, NY, USA, 181–184.  https://doi.org/10.1145/3366424.3383536
  * Han et al. (2023) Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2023\.  SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2023.acl-long.647
  * Harris (1954) Zellig S. Harris. 1954\.  Distributional Structure.  _WORD_ 10, 2-3 (1954), 146–162.  https://doi.org/10.1080/00437956.1954.11659520
  * He et al. (2023) Tianxing He, Jingyu Zhang, Tianle Wang, et al. 2023\.  On the Blind Spots of Model-Based Evaluation Metrics for Text Generation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2023.acl-long.674
  * Huang et al. (2018) Jiaji Huang, Yi Li, Wei Ping, and Liang Huang. 2018\.  Large Margin Neural Language Model. In _Proc. of EMNLP_. Association for Computational Linguistics, Brussels, Belgium, 1183–1191.  https://doi.org/10.18653/v1/D18-1150
  * Huang et al. (2019) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2019.  Challenges in Building Intelligent Open-domain Dialog Systems.
  * Huang et al. (2020) Po-Sen Huang, Huan Zhang, Ray Jiang, et al. 2020\.  Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 65–83.  https://doi.org/10.18653/v1/2020.findings-emnlp.7
  * Jagfeld et al. (2018) Glorianna Jagfeld, Sabrina Jenne, and Ngoc Thang Vu. 2018\.  Sequence-to-Sequence Models for Data-to-Text Natural Language Generation: Word- vs. Character-based Processing and Output Diversity. In _Proceedings of the 11th International Conference on Natural Language Generation_. Association for Computational Linguistics, Tilburg University, The Netherlands, 221–232.  https://doi.org/10.18653/v1/W18-6529
  * Jawahar et al. (2020) Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020\.  Automatic Detection of Machine Generated Text: A Critical Survey. In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Barcelona, Spain (Online), 2296–2309.  https://doi.org/10.18653/v1/2020.coling-main.208
  * Jelinek et al. (2005) F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. 2005\.  Perplexity—a measure of the difficulty of speech recognition tasks.  _The Journal of the Acoustical Society of America_ 62, S1 (2005), S63.  https://doi.org/10.1121/1.2016299