#  AMRFact: Enhancing Summarization Factuality Evaluation
with AMR-driven Training Data Generation

Haoyi Qiu$\dagger$ Kung-Hsiang Huang$\sharp$ Jingnong
Qu$\dagger$11footnotemark: 1 Nanyun Peng$\dagger$
$\dagger$University of California, Los Angeles $\sharp$University of Illinois
Urbana-Champaign
$\dagger${haoyiqiu, andrewqu2000<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Ensuring factual consistency is crucial in various natural language processing
tasks, particularly in abstractive summarization, where preserving the
integrity of information is paramount. Prior entailment-based approaches often
generate factually inconsistent summaries and then train a classifier on the
generated data. However, summaries produced by these approaches are either of
low coherence or lack error-type coverage. To address these issues, we propose
AMRFact, a novel framework that generates factually inconsistent summaries
using Abstract Meaning Representation (AMR). Our approach parses factually
correct summaries into AMR graphs and injects controlled factual
inconsistencies to create negative examples, allowing for coherent factually
inconsistent summaries to be generated with high error-type coverage.
Additionally, we present a data selection module NegFilter based on natural
language inference and BARTScore to ensure the quality of the generated
negative samples. Experimental results demonstrate that our approach
significantly outperforms previous systems on the AggreFact-FtSota dataset,
showcasing its efficacy in assessing factuality in abstractive summarization.

##  1 Introduction

Figure 1: Overview of AMRFact training phase. The generation module first
applies text-to-AMR models to the reference summaries and translates them into
AMR graphs. A generation model injects factual errors usually observed in
current state-of-the-art summarization models into AMR graphs and outputs
manipulated AMR graphs containing factually inconsistent information. We then
back-translate manipulated AMR graphs into summary texts to be used as
negative examples for training the text-based factuality evaluator. To ensure
quality, a selection module filters potential relatively low-quality negative
examples by combining NLI scores and BARTScore. In the end, utilizing our
automatically generated training data, we fine-tune a RoBERTa-based NLI model
to serve as the evaluation metric. This model interprets the original document
as the premise and the summary sentence as the hypothesis, employing the
predicted probability of entailment to determine factuality scores.

The recent progress in conditional text generation, fueled by the availability
of large-scale dataset, has led to models capable of generating highly fluent
abstractive summaries Narayan et al. (2018); Liu and Lapata (2019); Zhang et
al. (2019); Raffel et al. (2019); Lewis et al. (2020). However, previous
studies have shown that pretrained language models can produce summaries with
factually inconsistencies in over 70% of cases Goodrich et al. (2019);
Kryscinski et al. (2020); Pagnoni et al. (2021).

The concept of “factuality,” also known as factual consistency or
faithfulness, is pivotal in this context. As defined by Saurí and Pustejovsky
(2012) and Pustejovsky and Saurí (2008), factuality assesses whether the
content in a summary accurately reflects the facts, possibilities, or
situations presented in the source document. In the realm of summarization
task, the source document is considered as the ground truth, and an output
summary should align accurately and consistently with the facts of the source.
As factually consistent summaries are essential in preserving the integrity of
information dissemination, there is a pressing need to develop reliable
evaluation methods to assess the factual consistency of summarization systems.

Recent work has introduced the task of recognizing factually inconsistent
summaries as a downstream application of natural language inference (NLI) or
entailment recognition. In this paradigm, a hypothesis (generated summary) is
classified as either entailed, neutral, or contradicted by a given premise
(source document) Condoravdi et al. (2003); Dagan et al. (2005); Bowman et al.
(2015). The most common approach to learning an entailment-based factual
consistency metric is by generating synthetic data and training a classifier
on these generated data. Kryscinski et al. (2020) treats reference summaries
from CNN/DM Hermann et al. (2015) as positive samples and applies entity-
centric perturbations, such as entity replacement, to the reference summaries
to create negative samples (i.e., summaries factually inconsistent with the
source document). However, such an approach often produces sentences with poor
coherence caused by the string replacement operations. Goyal and Durrett
(2021) and Utama et al. (2022) addresses the coherence issue by training a
paraphrasing model and a text-infilling model to produce data. Yet, these
approaches cannot ensure the production of specific types of factual errors,
illustrated in Pagnoni et al. (2021), during the paraphrasing and text-
infilling processes. This means that models trained on data produced by these
methods may not reliably detect particular types of factual inconsistency in
summaries. We refer to this limitation as a lack of “error-type coverage” in
the generated data. Additionally, all prior methods lack a verification step
to validate the quality of the generated data, which may result in poorer
performance in detecting factual inconsistency.

Motivated by these challenges, we propose AMRFact, a novel framework that
generates coherent negative examples with high error-type coverage.
Concretely, our data generation module leverages Abstract Meaning
Representation (AMR) Banarescu et al. (2013) to introduce semantic-level
perturbations, creating negative examples. AMRs are intended to capture the
meaning of a sentence by abstracting away irrelevant syntactic features.
Consequently, using AMR enables us to generate more coherent summaries without
compromising the error-type coverage.

In detail, AMRFact starts with parsing factually consistent summaries into
semantic AMR graphs, and then injects factual inconsistencies that are
commonly observed in state-of-the-art summarization systems into the AMR
graphs Ghazarian et al. (2022); Ribeiro et al. (2022). These perturbed AMR
graphs are subsequently translated back into text summaries to serve as our
negative examples using a controllable generation model. Then, we devise a
novel selection module NegFilter to exclude invalid negative samples from our
training data. A valid negative summary must satisfy two criteria: (1) it must
not be directly inferable from the original summary and (2) it should not
stray significantly from the main topic of the document. We employ sentence-
level NLI and BARTScore to ensure compliance with each criterion,
respectively. Finally, a RoBERTa Liu et al. (2019) model is trained on the
created dataset along to distinguish factually consistent and inconsistent
summaries as the evaluation metric. Figure 1 offers an overview of our
proposed framework. The contributions of this work are summarized as follows:

  * •

We propose to use AMR-based perturbations to generate factually inconsistent
summaries, which allows for more coherent generation while maintaining high
error-type coverage.

  * •

We devise a novel negative data selection module to filter out invalid
negative summaries to validate the quality of generated data. This method
shows effectiveness and generalizability across various existing datasets.

  * •

Our approach achieves state-of-the-art performance on the AggreFact-FtSota
dataset, comprising news articles and summaries from CNN/DM and XSum Narayan
et al. (2018).

##  2 Related Work

###  2.1 Factual Consistency Metrics

Research on factual consistency metrics can be broadly classified into QA-
based and entailment-based approaches. QA-based metrics generally leverage
question generation (QG) and answering (QA) components to evaluate factual
consistency through the comparison of information units taken from summaries
and their original sources Wang et al. (2020); Scialom et al. (2021); Fabbri
et al. (2022). Entailment-based methods predict whether a summary is entailed
by its source article with document-sentence entailment models trained on
either synthetic Kryscinski et al. (2020); Yin et al. (2021); Utama et al.
(2022) or human-annotated data Goyal and Durrett (2021); Ribeiro et al.
(2022); Chan et al. (2023). Besides, Gekhman et al. (2023) propose a method
for generating synthetic data by annotating diverse model-generated summaries
using a LLM. Alternatively, Laban et al. (2022) break documents and summaries
into sentences and employ traditional sentence-level NLI models. Moreover,
Feng et al. (2023) utilize facts extracted from external knowledge bases to
improve the generalization across domains. Our approach employs an entailment-
based method, capitalizing on AMR-based perturbations to optimize error-type
coverage and synthetic data quality. We further improve data quality with a
negative data selection module, harnessing NLI and BARTScore.

###  2.2 AMR-based Evaluation

Few studies have attempted to employ AMR for evaluation. Ribeiro et al. (2022)
parse both the input documents and summaries into AMR graphs and train a
classifier to identify erroneous edges in the summary graph. However, their
approach relies on human-annotated data for effective training signals. On the
contrary, we parse summaries into AMR graphs and create negative training data
by applying perturbations to the graphs and converting the perturbed graphs
back to summaries. Ghazarian et al. (2022) uses a similar approach to generate
training data for a dialogue coherence evaluation metric. Notably, their
perturbation techniques focused on coherence evaluation, and thus ill-suited
for factual consistency evaluation. By contrast, we design our perturbation
method by taking inspiration from common factual errors in summaries produced
by state-of-the-art summarization systems.

##  3 Background

##### Abstract Meaning Representation.

Initially introduced by Banarescu et al. (2013), AMR is a representation
language, distilling text beyond surface syntax. AMR graphs effectively
capture the essence of texts by encoding abstract-level semantic information –
ranging from named entities and negations to coreferences and modalities. This
intrinsic capability positions AMR as a critical asset for various semantic-
related NLP tasks, such as summarization Liao et al. (2018) and machine
translation Song et al. (2019). In our study, we harness the potential of AMR
in the summarization factuality evaluation task by perturbing the graphs of
factually consistent summaries. Each perturbation reflects a factual error in
summarization.

Figure 2: AMR representations of a summary sentence.

In AMR graphs, nodes symbolize entities and concepts, interconnected through a
variety of relational edges. The concepts in AMR are often represented as
either words or keywords from the PropBank framesets Kingsbury and Palmer
(2002). These framesets, equipped with predefined arguments, enable a seamless
abstraction from syntactic constructs. An example of an AMR graph for a
summary is illustrated in Figure 2.

##  4 AMRFact

The process of generating summaries that are factually inconsistent with the
source document in AMRFact involves two primary states: introducing summary-
level perturbations based on AMR and filtering invalid negative data examples.
These stages are discussed in more detail in the following sections, which are
dedicated to explaining how factual errors are introduces (Section 4.1) and
the criteria for choosing negative examples (Figure 3).

Error Type |  Description |  Example
---|---|---
Predicate Error |  The predicate of the summary does not align with the information provided in the original document. |  Hingis ended a two-year ban after testing positive for cocaine at 2007 Wimbledon.
Entity Error |  The primary arguments, or their attributes, associated with the predicate are incorrect. |  Kournikova began a two-year ban after testing positive for cocaine at Wimbledon in 2007.
Circumstance Error |  The supplementary details, such as location or time, that define the context of a predicate are incorrect. |  Hingis began a two-year ban after testing positive for cocaine at Wimbledon in 2014.
Discourse Link Error |  Errors arise from the improper connection of statements within the discourse, such as errors in temporal ordering or causal link. |  Hingis began a two-year ban before testing positive for cocaine at Wimbledon in 2007.
Out of Article Error |  The statement conveys information that is absent in the original document. |  Hingis and Kournikova will team up again to play at this year’s All England Club.
Table 1: Typology of factual errors. Original document fragment for the
examples: (CNN) – Martina Hingis and Anna Kournikova will team up again to
play at this year’s Wimbledon championships. […] Swiss star Hingis won five
grand slam singles crowns and nine in doubles during a glittering career which
ended under a cloud in 2007, when she was suspended for two years for testing
positive for cocaine at Wimbledon. […]

###  4.1 AMR-based Summary Perturbations

Our approach targets five specific types of factual inconsistencies: predicate
error, entity error, circumstance error, discourse link error, and out of
article error, as categorized by Pagnoni et al. (2021). A detailed description
of these factual errors is presented in Table 1. The first four types of
errors can be generated by perturbing the AMR graph of a summary that is
initially factually consistent, and then reconverting the perturbed graph back
into a natural language summary. Specifically, we start with a source document
$D$ alongside its factually consistent summary $S^{+}$, which is then parsed
into an acyclic AMR graph $G^{+}$. This graph is manipulated to form a new AMR
graph $G^{-}$ that embodies factual inconsistencies. For instance, as
illustrated in Figure 1, we produce a predicate error by swapping the original
predicate “work” to “leisure”. The perturbed AMR graph $G^{-}$ is then
transformed into a factually inconsistent natural language summary $S^{-}$
using an AMR-to-text model Ribeiro et al. (2021). On the other hand, out of
article error can be introduced by adding new information or substituting
existing content in the summary.

##### Predicate Error.

Predicate errors occur when the predicate in a summary does not align with the
information in the source document. To simulate this type of error, we modify
part of the original AMR graph by incorporating a sub-graph that contains
factually incorrect information. These inconsistencies primarily emerge from
two prcoesses: (1) by adding or removing polarity, which alters the original
statement’s truth value; and (2) through the substitution of a predicate with
its antonym, which directly contracits the original meaning. The capability of
the AMR-to-Text model is particularly highlighted when it efficiently handles
complex modifications, such as integrating the word “to” in conjunction with
verb replacements, thereby maintaining the coherence of the summary.

##### Entity Error.

Entity errors manifest when the entities associated with a predicate in a
summary are incorrectly attributed or erroneous, despite the original text
correctly representing the relationship. These errors typically stem from two
principal sources: (1) by swapping the roles of the agent and the patient,
which results in misattribution of actions or characteristics; and (2) through
the substitution of specific entity, such as names, numbers, and date, which
leads to inaccuracies in the detailed information presented in the summary.
When these replacements are made, the AMR-to-text model adeptly modifies the
corresponding segments of the summary. This ensures that the output remains
grammatically coherent and naturally articulated. For example, if an agent and
a patient are interchanged, the model will also adjust the nested subgraphs
linked to these arguments to maintain narrative consistency.

##### Circumstance Error.

Circumstance errors in summaries emerge when there is incorrect or misleading
information regarding the context of predicate interactions, specifically in
terms of location, time, and modality. These errors mainly originate from two
areas: (1) by intensifying the modality, which alters the degree of certainty
or possibility expressed in the statement; and (2), by substituting specific
entities like locations and times, leading to inaccuracies in contextual
details. The use of an AMR-to-Text model is particularly advantageous in these
scenarios. It facilitates verb adaptations that correspond to changes in
modality, thereby improving the natural flow and coherence of the summary
despite the shift in context.

##### Discourse Link Error.

Discourse link errors pertain to mistakes in the logical connections between
various statements in a summary, often manifesting as incorrect temporal
sequencing or flawed discourse relations. Specifically, we focus on two key
types of discourse links: (1) temporal order, which deals with the sequence of
events, and (2) causality, which pertains to the cause-and-effect
relationships between statements. By employing the AMR-to-Text model, we
ensure that despite these deliberate perturbations, the resulting summary
remains both coherent and grammatically sound.

##### Out of Article Error.

Summaries are expected to contain only information that can be inferred from
the source document, and deviations from this rule need to be clearly
identified. To create an “out of article” error, one could follow a similar
method as previously discussed, involving alterations in entities, times, or
locations. However, in this instance, we intentionally introduce vocabulary
not present in the original document. The effectiveness of the AMR-to-Text
model is particularly evident in this scenario, as it adeptly incorporates
additional elements, such as the word “with” alongside new verb substitutions.
This approach ensures that the summary, despite its factual inaccuracies,
maintains a natural and fluent narrative flow.

###  4.2 Invalid Negative Data Filtering

Figure 3: An example showing how our invalid negative data filtering module
works. In the above three examples, only the first perturbed summary is valid
since both of its entailment score and BARTScore satisfy the criteria
described in Figure 3.

In our data generation process, we noticed that certain generated negative
examples either blatantly contradicted the source article’s facts or failed to
modify the semantics of the reference summary, inadvertently becoming positive
examples. These were deemed as invalid negative examples. We hypothesize that
training detection models on such invalid negative data could potentially
impair their performance. To address this issue, we introduce a module
NegFilter specifically designed for filtering out invalid negative data.

A perturbed summary is considered valid only if it satisfy two essential
criteria: (1) It must be sufficiently distint from the original summary to
avoid being mistaken for a mere variation, thereby ensuring it iss not
misclassified as a positive example; and (2) despite the introduced
perturbations, it should maintain a discernible connection to the source
document, without diverging excessively.

To ensure the first criterion – distinctiveness of generated summaries – we
utilize sentence-level NLI inspired by previous studies Wan and Bansal (2022);
Huang et al. (2023a, b). Concretely, we employ a RoBERTa large model fine-
tuned on the MNLI corpus Liu et al. (2019); Williams et al.
(2018)111https://huggingface.co/roberta-large-mnli to quantify the entailment
score between an original summary ($S^{+}$) and its perturbed counterpart
($S^{-}$). Then, perturbed summaries that score exceed an empirically selected
threshold ($\tau_{1}$) in entailment scores are discarded, given their
elevated likelihood of being inferred from $S^{+}$.

For the second criterion – maintaining relevance to the source document – we
propose to use BARTScore Yuan et al. (2021), fine-tuned on the CNN/DM dataset
Hermann et al. (2015), to assess the semantic alignment between the perturbed
summary ($S^{-}$) and its corresponding source document ($D$). Similarly,
perturbed summaries with a BARTScore falling below the empirically determined
threshold ($\tau_{2}$) are excluded due to their divergence from the source
documents.

In summary, our proposed high-quality negative examples selection module
NegFilter will take a source document ($D$), an original summary ($S^{+}$),
and a perturbed summary ($S^{-}$). Only when both criteria are satisfied, the
generated factually inconsistent summaries ($S^{-}$) will be included into our
negative training data. Formally,

$\displaystyle\mathcal{M}(D,S^{+},S^{-})=\textbf{True}\text{ if and only if
}\leavevmode\nobreak\ $
$\displaystyle\mathbbm{1}[\mathcal{N}(S^{+},S^{-})<\tau_{1}]\cdot\mathbbm{1}[\mathcal{B}(D,S^{-})>\tau_{2}]=1,$
(1)

where $N$ represents the entailment score, and $B$ denotes BARTScore. Figure 3
provides an illustration of our proposed selection module222We set
$\tau_{1}=0.9$ and $\tau_{2}=-2$ for best performance..

###  4.3 Detecting Factual Inconsistency

To learn to detect factual inconsistency, we fine-tune a roberta-large model
pre-trained on the MNLI corpus on our augmented AMRFact dataset, given that
this model has been fine-tuned on a relevant task. The input to our model is a
concatenation of the document premise $D$ with a summary hypothesis ($S^{+}$
or $S^{-}$). Although our output linear layer performs a three-way
classification, we focus only on the “entailment” and “contradiction” outputs.
The training data paired with the reference summaries are labeled as
“entailment”, whereas those coupled with the perturbed summaries are labeled
as “contradiction”. This approach allows for a more nuanced understanding of
the relationship between the document content and its summaries,
distinguishing factual consistency from inconsistency.

##  5 Experimental Settings

Our experiments aim to evaluate the efficacy of AMRFact-generated sentence-
level negative examples for enhancing NLI dataset augmentation. We assess the
downstream performance of the proposed NLI model by evaluating it against
several benchmarks for identifying the factual inconsistencies within
generated summaries. Section 5.1 outlines our training and evaluation
datasets, providing a comprehensive overview of the data used in our
experiments. In Section 5.2, we introduce the baseline models against which
our proposed method will be compared, setting the stage for a thorough
evaluation of our approach’s effectiveness in enhancing NLI model performance.

###  5.1 Datasets

####  5.1.1 Training Dataset

We utilized the AMRFact data generation pipeline to the training split of the
CNN/DM corpus Hermann et al. (2015), a dataset initially designed for
question-answering and later repurposed for summarization by Nallapati et al.
(2016). This dataset compromises English news articles and their associated
multi-sentence human-generated summaries. For our study, we modify the data
such that each instance pairs a document with a single-sentence reference
summary. Subsequent to this, candidate perturbed summaries are generated and
processed by our invalid negative data filtering module NegFilter. The
criteria for retention of these summaries are their strong semantic congruence
with the source documents, while ensuring they are distinct from the original
reference summaries. These selectively chosen perturbed summaries are combined
with the positive examples (reference summaries) to form our NLI training
dataset. This enriched dataset is aimed at bolstering the NLI model’s capacity
to discern factual consistency in summary generation.

####  5.1.2 Evaluation Dataset

| Polytope | SummEval | FRANK | Wang’20 | CLIFF | Goyal’21 | Cao’22 | Total
---|---|---|---|---|---|---|---|---
Cnn | val | 34 | 200 | 75 | - | 150 | - | - | 459
test | 34 | 200 | 175 | - | 150 | - | - | 559
Xsum | val | - | - | - | 120 | 150 | 50 | 457 | 777
test | - | - | - | 119 | 150 | 50 | 239 | 558
Table 2: Statistics of AggreFact-Cnn-FtSota and AggreFact-Xsum-FtSota. These
two subsets consist of Polytope Huang et al. (2020), SummEval Fabbri et al.
(2021), FRANK Pagnoni et al. (2021), Wang’20 Wang et al. (2020), CLIFF Cao and
Wang (2021), Goyal’21 Goyal and Durrett (2021), Cao’22 Cao et al. (2022).

Tang et al. (2023) introduced the benchmark AggreFact, which consolidates nine
existing datasets focused on factuality for a finer-grained comparison of
factuality assessment systems. They stratify the benchmark based on the
underlying summarization model, categorized into FtSota, EXformer and Old
according to their development timeline. FtSota encompasses state-of-the-art
fine-tuned summarization models, such as BART Lewis et al. (2020), PEGASUS
Zhang et al. (2019) and T5 Raffel et al. (2019). EXformer represents early
Transformer-based models, including notable ones like BERTSum (Liu and Lapata,
2019) and GPT-2 (Radford et al., 2019). Models like Pointer-Generator See et
al. (2017) and BottomUp Gehrmann et al. (2018) are classified as Old. In line
with Tang et al. (2023)’s recommendations, our performance reporting focuses
on the FtSota split, as it most accurately reflects the challenges of
unfaithfulness in the most advanced summarization models. Table 2 shows the
statistics of AggreFact-FtSota.

###  5.2 Baseline Models

##### FactCC

Kryscinski et al. (2020) is an entailment-based metric that uses binary
classification to evaluate summary factuality. It’s trained on synthetic data
created through rule-based transformations of source document sentences.

##### DAE

Goyal and Durrett (2021) propose an arc entailment approach that evaluates the
factuality of each dependency arc of the generated summary independently with
respect to the input article and then uses their aggregation as the overall
score. Unlike previous work that report the performance of DAE trained on
human-annotated data, we opt for DAE-Ent, a variant of DAE trained on
synthetic data, for fair comparison.

##### QuestEval

Scialom et al. (2021) introduces a QA-based metric that calculates factuality
by aggregating answer overlap scores from queries derived from both the input
article and the summary.

##### SummaC

Laban et al. (2022) focuses on detecting factual inconsistencies by
aggregating sentence-level entailment scores between the input document and
summary sentences. It offers two variants, SummaC-ZS and SummaC-Conv, which
differ in their aggregation methods.

##### QAFactEval

Fabbri et al. (2022) is a QA-based metric analogous to the precision-based
component of QuestEval and includes optimized question answering, generation,
and answer-overlap components.

##### AlignScore

Zha et al. (2023) provides a comprehensive metric based on information
alignment between two texts. It integrates a diverse range of data sources
into its training framework, resulting in 4.7M training examples from 7 tasks.

##### FalseSum

Utama et al. (2022) enhances NLI data with controllable text generation for
improved factuality evaluation. It generates factually inconsistent summaries
by predicting masked span of reference summaries with control code.

##### ChatGPT-ZS

Luo et al. (2023) uses a zero-shot template and directly asks for a binary
label of summary factuality.

##### ChatGPT-CoT

Luo et al. (2023) also uses a zero-shot template but invokes chain-of-thought
(CoT) style reasoning in its prompt. Similar to ChatGPT-ZS, it directly asks
for a binary factuality label for a given summary.

##### ChatGPT-DA

Wang et al. (2023) uses a direct assessment (DA) prompt template that asks to
assign a factual consistency score to a summary on a continuous scale from 0
to 100.

##### ChatGPT-Star

Wang et al. (2023) is an alternative version of ChatGPT-DA that asks LLMs to
score summaries on a scale of one-to-five.

##### G-Eval

Liu et al. (2023) is a framework of using large language models with chain-of-
thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG
outputs. We implement this with GPT-4 Turbo (gpt-4-1106-preview).

For fair comparison with entailment-based models, we retrain FactCC and
FalseSum with robera-large-mnli as the base models.

###  5.3 Evaluation Criteria

We use balanced accuracy Brodersen et al. (2010) as our evaluation metric,
which requires models to produce a scalar value instead of a binary label.
Balanced accuracy is defined as:

$BA_{cc}=\frac{1}{2}(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}),$ (2)

where TP stands for true positive, FP false positive, TN true negative, and FN
false negative. Moreover, we set the threshold for each model such that the
threshold ($\tau$) optimizes its performance on the validation set, following
Laban et al. (2022) and Tang et al. (2023). Formally, given our model
$\mathcal{E}$ and a source document $D$ with a candidate summary $S$:

$S\text{ is factually {consistent} with }D\text{ if }\mathcal{E}(D,S)>\tau.$
(3)

##  6 Results and Discussion

###  6.1 Main Results

AggreFact- Cnn-FtSota AggreFact- XSum-FtSota Avg DAE 65.0 $\pm$ 3.5 62.3 $\pm$
1.9 63.7 QuestEval 70.2 $\pm$ 3.2 59.5 $\pm$ 2.7 64.9 SummaC-ZS 64.0 $\pm$ 3.8
56.4 $\pm$ 1.2 60.2 SummaC-Conv 61.0 $\pm$ 3.9 65.0 $\pm$ 2.2 63.0 QAFactEval
67.8 $\pm$ 4.1 63.9 $\pm$ 2.4 65.9 FactCC 57.6 $\pm$ 3.9 57.2 $\pm$ 1.7 57.4
FalseSum 50.5 $\pm$ 3.3 54.7 $\pm$ 1.9 52.6 AlignScore 67.0 $\pm$ 3.1 60.3
$\pm$ 1.9 63.7 ChatGPT-ZS 56.3 $\pm$ 2.9 62.7 $\pm$ 1.7 59.5 ChatGPT-COT 52.5
$\pm$ 3.3 55.9 $\pm$ 2.1 54.2 ChatGPT-DA 53.7 $\pm$ 3.5 54.9 $\pm$ 1.9 54.3
ChatGPT-Star 56.3 $\pm$ 3.1 57.8 $\pm$ 0.2 57.1 G-Eval 69.9 $\pm$ 3.5 65.8
$\pm$ 1.9 67.9 AMRFact (ours) 72.3 $\pm$ 2.5 64.1 $\pm$ 1.8 68.2

Table 3: Balanced binary accuracy (%) on the AggreFact-FtSota test set. We
show 95% confidence intervals. Highest performance is highlighted in bold. The
Avg score is computed by taking the arithmetic average of the performance on
AggreFact-Cnn-FtSota and AggreFact-XSum-FtSota.

The summarized results in Table 3 indicate that our model AMRFact sets a new
benchmark on the AggreFact-FtSota test set. We report the performance on
CNN/DM and XSum using separate thresholds, following Tang et al. (2023).
AMRFact establishes a new state-of-the-art score of 72.3% on the CNN/DM split,
outperforming the nearest competitor by a margin of 2.1%. On XSum split, our
model’s performance is competitive, tying with the highest score at 64.1%.
With an overall average score of 68.2% across both datasets, we outperform the
previous best model in the average by 2.3%. The improvements underscore the
advantage of incorporating AMR in generating training data and the strategic
selection of high-quality data through our data selection module.

###  6.2 Impact of Our Filtering Module

To validate the effectiveness and generalizability of our invalid negative
data filtering component NegFilter, we apply this module to the data generated
by FactCC and FalseSum and re-train a RoBERTa on these newly filtered data.
Additionally, we train another RoBERTa on our generated data without the
filtering component. The results are summarized in Table 4. We observe a
substantial performance gain when all three data generation methods
incorporate the proposed invalid data filtering module. Specifically, FactCC’s
balanced accuracy improves from 57.6% to 67.9% on the CNN/DM and from 57.2% to
63.8% on XSum, while FalseSum sees an enhancement from 50.5% to 52.3% on
CNN/DM and from 54.7% to 59.7% on XSum. These improvements affirm the
effectiveness and generalizability of our approach in filtering invalid
negative training data. Conversely, removing the filtering from our AMRFact
data causes a decrease in balanced accuracy, dropping from 72.3% to 63.1% on
CNN/DM and from 64.1% to 57.6% on XSum, resulting in an average drop from
68.2% to 60.4%. This stark contrast underlines the critical role of our
invalid negative data filtering module in enhancing the performance of
inconsistency detection models, supporting our hypothesis mentioned in Figure
3 that training inconsistency detection models on invalid negative data hurts
the performance.

AggreFact- Cnn-FtSota AggreFact- XSum-FtSota Avg FactCC 57.6 $\pm$ 3.9 57.2
$\pm$ 1.7 57.4 \+ Filtering 67.9 $\pm$ 2.3 63.8 $\pm$ 2.2 65.8 $\Delta$: +10.3
$\Delta$: +6.6 FalseSum 50.5 $\pm$ 3.3 54.7 $\pm$ 1.9 52.6 \+ Filtering 52.3
$\pm$ 1.8 59.7 $\pm$ 2.2 56.0 $\Delta$: +1.8 $\Delta$: +5.0 AMRFact (ours)
72.3 $\pm$ 2.5 64.1 $\pm$ 1.8 68.2 \- Filtering 63.1 $\pm$ 3.0 57.6 $\pm$ 2.0
60.4 $\Delta$: -9.2 $\Delta$: -6.5

Table 4: Balanced binary accuracy (%) on the AggreFact-Cnn-FtSota and
AggreFact-XSum-FtSota test set with or without our invalid negative data
filtering module.

###  6.3 Remaining Challenges

Figure 4: Distribution of the errors that our model fails to identify. The
most dominant type of error is the annotation error. And errors stemming from
out-of-article content were another significant source of inaccuracies.

To gain deeper insights into the limitations and remaining challenges for our
model, we conduct a detailed manual analysis. This involved examining a random
sample of 50 examples from AggreFact-FtSota where our model incorrectly
labeled factually inconsistent instances. The results of this analysis are
meticulously detailed in Figure 4. From this thorough assessment, several key
findings emerged. First, the most dominant type of error (30$\%$) is the
annotation error. This underscores the challenges in creating completely
reliable benchmarks through crowd-sourced methods, a complexity also noted in
the work of Laban et al. (2023). Second, errors stemming from out-of-article
content were another significant source of inaccuracies. Intriguingly, upon
further investigation, we discovered that a notable portion (8 out of 14) of
these out-of-article errors were, in fact, factually accurate. For instance,

> Former Wales captain Martyn Williams says Dan Biggar’s decision to sign a
> new four-year contract with Ospreys will benefit the region.

The above summary was labeled as unfaithful because the corresponding source
document (see Table 5) does not mention that Martyn Williams was a captain.
Nonetheless, this information is factually correct, as evidenced by a
Wikipedia search. This discrepancy suggests that our model, in this case,
might have utilized its parametric knowledge learned during the pre-training
phase of RoBERTa rather than the content of the article itself. To mitigate
this issue, future research efforts can look into creating negative training
examples that consist of out-of-article errors which, although truthful
according to worldly facts, are unfaithful to the source document. This
approach could help fine-tune the model’s ability to distinguish between
general knowledge and the specific information presented in a given text.

##  7 Conclusion

We introduce a novel framework, AMRFact, which leverages Abstract Meaning
Representation (AMR) to generate factually inconsistent summaries for
summarization factuality evaluation. Our method addresses the prevalent issues
of low coherence and insufficient error-type coverage observed in prior
entailment-based approaches. By parsing factually consistent reference
summaries into AMR graphs and injecting controlled factual inconsistencies, we
successfully create coherent, but factually inconsistent summaries with broad
error-type coverage. The introduction of the NegFilter module, which utilizes
natural language inference and BARTScore, further enhances the quality of the
generated negative samples, filtering out invalid summaries. The effectiveness
and generalizability of this approach are validated through its state-of-the-
art performance on the AggreFact-FtSota dataset. Our contributions not only
advance the field in generating high-quality, factually inconsistent summaries
but also provide a scalable and efficient solution for enhancing the
factuality evaluation of summarization systems. The results underscore the
potential of AMR-based perturbations in improving the integrity and
reliability in natural language generation.

## References

  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013.  Abstract meaning representation for sembanking.  In _LAW@ACL_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Brodersen et al. (2010) Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M. Buhmann. 2010.  The balanced accuracy and its posterior distribution.  _2010 20th International Conference on Pattern Recognition_ , pages 3121–3124.
  * Cao et al. (2022) Meng Cao, Yue Dong, and Jackie Cheung. 2022.  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3340–3354, Dublin, Ireland. Association for Computational Linguistics.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chan et al. (2023) Hou Pong Chan, Qi Zeng, and Heng Ji. 2023.  Interpretable automatic fine-grained inconsistency detection in text summarization.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 6433–6444, Toronto, Canada. Association for Computational Linguistics.
  * Condoravdi et al. (2003) Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003.  Entailment, intensionality and text understanding.  In _Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning_ , pages 38–45.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges Workshop_.
  * Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022.  QAFactEval: Improved QA-based factual consistency evaluation for summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2587–2601, Seattle, United States. Association for Computational Linguistics.
  * Fabbri et al. (2021) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  Summeval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Feng et al. (2023) Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023.  Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge.  _arXiv preprint arXiv:2305.08281_.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Gekhman et al. (2023) Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023.  Trueteacher: Learning factual consistency evaluation with large language models.  _ArXiv_ , abs/2305.11171.
  * Ghazarian et al. (2022) Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022.  DEAM: Dialogue coherence evaluation using AMR-based semantic manipulations.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 771–785, Dublin, Ireland. Association for Computational Linguistics.
  * Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Mohammad Saleh, and Peter J. Liu. 2019.  Assessing the factual accuracy of generated text.  _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462, Online. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems_ , December 7-12, 2015, Montreal, Quebec, Canada, pages 1693–1701.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Huang et al. (2020) Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020.  What have we achieved on text summarization? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 446–469, Online. Association for Computational Linguistics.
  * Huang et al. (2023a) Kung-Hsiang Huang, Hou Pong Chan, and Heng Ji. 2023a.  Zero-shot faithful factual error correction.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5660–5676, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023b) Kung-Hsiang Huang, Kathleen McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. 2023b.  Faking fake news for real fake news detection: Propaganda-loaded training data generation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14571–14589, Toronto, Canada. Association for Computational Linguistics.
  * Kingsbury and Palmer (2002) Paul Kingsbury and Martha Palmer. 2002.  From TreeBank to PropBank.  In _Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02)_ , Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA).
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Laban et al. (2023) Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander R Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023.  Llms as factual reasoners: Insights from existing benchmarks and beyond.  _arXiv preprint arXiv:2305.14540_.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. 2022.  Summac: Re-visiting nli-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liao et al. (2018) Kexin Liao, Logan Lebanoff, and Fei Liu. 2018.  Abstract meaning representation for multi-document summarization.  _arXiv preprint arXiv:1806.05655_.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: Nlg evaluation using gpt-4 with better human alignment.  _ArXiv_ , abs/2303.16634.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023.  Chatgpt as a factual inconsistency evaluator for text summarization.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Pustejovsky and Saurí (2008) James Pustejovsky and Roser Saurí. 2008.  A factuality profiler for eventualities in text.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Ribeiro et al. (2022) Leonardo F. R. Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, and Mohit Bansal. 2022.  FactGraph: Evaluating factuality in summarization with semantic graph representations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3238–3253, Seattle, United States. Association for Computational Linguistics.
  * Ribeiro et al. (2021) Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021.  Investigating pretrained language models for graph-to-text generation.  In _Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI_ , pages 211–227, Online. Association for Computational Linguistics.
  * Saurí and Pustejovsky (2012) Roser Saurí and James Pustejovsky. 2012.  Are you sure that this happened? assessing the factuality degree of events in text.  _Computational Linguistics_ , 38(2):261–299.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  QuestEval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Song et al. (2019) Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019.  Semantic neural machine translation using AMR.  _Transactions of the Association for Computational Linguistics_ , 7:19–31.
  * Speer and Havasi (2012) Robyn Speer and Catherine Havasi. 2012.  Representing general relational knowledge in ConceptNet 5.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 3679–3686, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tang et al. (2023) Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023.  Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11626–11644, Toronto, Canada. Association for Computational Linguistics.
  * Utama et al. (2022) Prasetya Utama, Joshua Bambrick, Nafise Moosavi, and Iryna Gurevych. 2022.  Falsesum: Generating document-level NLI examples for recognizing factual inconsistency in summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2763–2776, Seattle, United States. Association for Computational Linguistics.
  * Wan and Bansal (2022) David Wan and Mohit Bansal. 2022.  FactPEGASUS: Factuality-aware pre-training and fine-tuning for abstractive summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1010–1028, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023.  Is chatgpt a good nlg evaluator? a preliminary study.  _ArXiv_ , abs/2303.04048.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Yin et al. (2021) Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021.  DocNLI: A large-scale dataset for document-level natural language inference.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4913–4922, Online. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27263–27277. Curran Associates, Inc.
  * Zha et al. (2023) Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023.  Alignscore: Evaluating factual consistency with a unified alignment function.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2019) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  _CoRR_ , abs/1912.08777.

##  Appendix A Error Typology Detailed Description

##### Predicate Error.

Predicate errors arise when a summary’s predicate is inconsistent with its
source document. We replace part of the original AMR with a factually
inconsistent sub-graph to model this error. This discrepancies stem from two
main mechanisms: (1) polarity addition or removal, and (2) antonym
substitution. In AMR graphs, the negation of a predicate is marked by a
polarity argument. By directly adding or removing this argument, we change the
negation in the sentence. Another approach is the antonym substitution. Here,
we replace the concepts with their antonyms that holding Antonym, NotDesires,
NotCapableOf, and NotHasProperty relations in ConceptNet Speer and Havasi
(2012), and therefore modify the sentence-level relations. The strengths of
the AMR-to-Text model become evident when adaptations, like incorporating to,
are coupled with verb substitutions to enhance summary coherence.

##### Entity Error.

Entity errors occur when the entities of a predicate are misattributed or
wrong, even if the original text correctly expressed the relation. Such errors
primarily arise from (1) agent-patient swap and (2) entity (i.e., names,
numbers, and date) substitution. In AMR graphs, the clear demarcation between
agent (ARG0) and patient (ARG1) allows for straightforward swaps. For named
entity modifications, we employ a rule-based approach using the SpaCy NER
tagger Honnibal et al. (2020) to extract the entities holding PERSON, ORG,
NORP, FAC, GPE, PRODUCT, WORK_OF_ART, EVENT, PERCENT, MONEY, QUANTITY, and
CARDINAL labels and then randomly select an entity from the summary to be
replaced by a different entity with the same label from the same source
document. After replacements, the AMR-to-text model effectively adjusts the
summary segments, ensuring outputs that are both grammatically correct and
naturally phrased. For instance, after swapping agent and patient, the nested
subgraphs related to the arguments will be adjusted accordingly.

##### Circumstance Error.

Beyond core arguments, predicates can be refined using attributes that
elucidate the context of their interactions, such as location, time, and
modality. Circumstance errors arise primarily from two sources: (1) modality
strengthening and (2) entity (i.e., locations and time) substitution. We
strengthen the modality by replacing the concepts that controls modals (i.e.,
permit-01, possible-01, likely-01, recommend-01, wish-01 $\rightarrow$
obligate-01). We emphasize modality strengthening over its inverse to avoid
generating truth-conditionally compatible sentences. For entity substitution,
we adopt a rule-based strategy with the SpaCy NER tagger Honnibal et al.
(2020) to extract the entities holding LOC, DATE, and TIME labels and then
randomly select an entity from the summary to be replaced by a different
entity with the same label from the same source document. The advantages of
the AMR-to-Text model become apparent such as it introduces verb adaptations,
enhancing the fluency of the summary due to the modality shift.

##### Discourse Link Error.

Discourse link errors refer to inaccuracies in the connections between
different statements. These include errors of incorrect temporal ordering or
incorrect discourse links between statements. We distinguish between two
primary discourse links: (1) temporal order and (2) causality. To manipulate
temporal ordering, we change before to after, after to before, and now to
either before or after by recognizing their presence in a time argument. For
causality modifications, we alter argument structures associated with
cause-01, either at the root or as a modifier, effectively reversing the
causal relationship. This includes transitions such as interchanging because
with therefore. Utilizing the AMR-to-Text model ensures that the generated
summary remains coherent and grammatically correct based on the perturbations.

##### Out of Article Error.

Given that summaries should encapsulate only inferable information from the
source document, it is imperative to categorize deviations from this
principle. To induce an out of article error, we could adopt the same approach
discussed before of changing entities or changing time and location. However,
here we will introduce vocabulary that is external to the original source
document. We propose the integration of irrelevant sources into AMR graphs by
selecting AMR elements like concepts, ops, and ARGs, and substituting them
with items from unrelated documents, coherence is deliberately disrupted. The
strength of the AMR-to-Text model is showcased in this context, as it
introduces adaptive elements, like with, in tandem with new verb replacements,
ensuring the summary retains fluency.

##### Generation.

Given the source document $D$ alongside its factually consistent summary
$S^{+}$, the generation module first applies text-to-AMR models to the summary
and translates it into directed and acyclic AMR graph $G^{+}$. Next, our
contronlable generation model $\mathcal{G}$ injects factual errors specified
above into AMR graph $G^{+}$, and outputs manipulated AMR graph $G^{-}$
containing factually inconsistent information. We then back-translate the
manipulated AMR graphs $G^{-}$ into summary text $S^{-}$ to be served as
negative examples for training the text-based factuality evaluator.

The 26-year-old fly-half has agreed a new deal which could keep him in Wales
until the 2019 World Cup. Williams, who played 100 times for Wales, believes
it will help Ospreys keep and recruit players. "It’s fabulous news for Welsh
rugby and the Ospreys in particular," he said on BBC Wales’ Scrum V TV
programme. "Not only is Dan committing himself to Wales for the next four
years, but it helps the Ospreys with recruitment for the next couple of
seasons. "If players were looking to sign and come to the Ospreys if they can
see Dan Biggar is going to be there for the next three or four seasons that
helps them as well." Biggar’s current deal was due to expire at the end of
this season. He is the first of Wales’ 17 dually contracted players to re-sign
on the deals which are 60% funded by the WRU and 40% by the region. In
addition to potentially attracting new players to the region, Biggar’s
decision to stay may help negotiations with his Ospreys and Wales colleagues
scrum-half Rhys Webb and second row Alun Wyn Jones. Both players’ contracts
expire in the summer of 2016, with Ospreys skipper Jones saying in November
that he was still weighing up his options. Scarlets are in talks with Wales
centre Scott Williams over extending his dual contract, and have secured the
return of British and Irish Lions centre Jonathan Davies from Clermont
Auvergne next season.
---
Table 5: The source document corresponding to the factual but unfaithful
summary mentioned in Section 6.3.