#  From Chaos to Clarity: Claim Normalization to Empower Fact-Checking

Megha Sundriyal$1$, Tanmoy Chakraborty$2$, Preslav Nakov$3$
$1$Indraprastha Institute of Information Technology Delhi
$2$Indian Institute of Technology Delhi
$3$Mohammed Bin Zayed University of Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With the proliferation of social media platforms, users are exposed to vast
information, including posts containing misleading claims. However, the
pervasive noise inherent in these posts presents a challenge in identifying
precise and prominent claims that require verification. Extracting the core
assertions from such posts is arduous and time-consuming. We introduce a novel
task, called Claim Normalization (aka ClaimNorm) that aims to decompose
complex and noisy social media posts into more straightforward and
understandable forms, termed _normalized claims_. We propose CACN, a
pioneering approach that leverages chain-of-thought and claim check-worthiness
estimation, mimicking human reasoning processes, to comprehend intricate
claims. Moreover, we capitalize on large language models’ powerful in-context
learning abilities to provide guidance and improve the claim normalization
process. To evaluate the effectiveness of our proposed model, we meticulously
compile a comprehensive real-world dataset, CLAN, comprising more than $6k$
instances of social media posts alongside their respective normalized claims.
Experimentation demonstrates that CACN outperforms several baselines across
various evaluation measures. A rigorous error analysis validates CACN’s
capabilities and pitfalls111We release our dataset and code at
https://github.com/LCS2-IIITD/CACN-EMNLP-2023.

##  1 Introduction

Online social media platforms have enabled a new paradigm in communication,
breaking down all geographical barriers and concocting unprecedented
opportunities for communication and knowledge exchange. However, this freedom
presents a growing threat to our society, with numerous harmful impacts
witnessed in crucial events such the $45th$ Presidential US Elections Allcott
and Gentzkow (2017), the COVID-19 Pandemic Rocha et al. (2021), the
Ukraine–Russia War Khaldarova and Pantti (2020), etc. False claims are an
intrinsic aspect of fabricated news, rumors, propaganda, and misinformation.

Figure 1: Illustration of our proposed Claim Normalization task, highlighting
the normalized claims authored by fact-checkers for social media posts from
distinct social media platforms.

Journalists and fact-checkers work tirelessly to validate such claims in
spoken or written form, sifting through this avalanche of digital information
to determine the truth. To further address this pressing issue, several
independent fact-checking organizations have emerged in recent years, such as
Snopes, FullFact, and PolitiFact, which play a crucial role in verifying the
accuracy of online content.222https://www.snopes.com, https://fullfact.org,
https://www.politifact.com However, the rate at which online information is
being disseminated far outpaces the capacity of fact-checkers, making it
difficult to verify every claim. This, in turn, leaves numerous unverified
claims circulating online, potentially reaching millions before they can be
verified.

While the complete automation of the fact-checking pipeline may pose hazards
to accountability and reliability, several recent studies have targeted
identifying downstream tasks suitable for automation, such as detecting claims
Daxenberger et al. (2017); Chakrabarty et al. (2019), evaluating their
worthiness for fact-checking Jaradat et al. (2018); Wright and Augenstein
(2020), and validating them by retrieving relevant shreds of evidence Zhi et
al. (2017); Hanselowski et al. (2018); Soleimani et al. (2020). In light of
the growing challenges faced by fact-checkers in verifying the accuracy of
social media claims, we propose the novel task of claim normalization. This
task aims to extract and simplify the central assertion made in a long, noisy
social media post. This novel paradigm can optimize fact-checkers’ efficacy
and curtail their workload while maintaining a high level of precision and
conscientiousness in fact-checking. We provide a more detailed explanation of
why the claim normalization task is essential and illustrate its significance
in Appendix A.1.

In our problem formulation, given an input social media claim, the system
needs to simplify it in a concise form containing the post’s central assertion
that fact-checkers can easily verify. To better understand our motivation, we
illustrate the task in Figure 1. The first social media post reads,
‘Cyanocobalamin is a synthetic form of Vitamin B12…If you’re on B12
supplements, throw them away.’ This post contains some extraneous information
that holds no relevance for fact-checkers. As a result, they distill the
information and summarize it as, ‘Cyanocobalamin, the most common form of
Vitamin B12, is toxic.’ Fact-checkers tasked with verifying the accuracy of
such noisy posts need to read through them and condense their content to
obtain a concise claim that can be easily fact-checked. Unfortunately, this
process can be exceedingly time-consuming for fact-checkers. By automating the
claim normalization process, fact-checkers can work more efficiently and
promptly provide accurate information to the public. This underscores the
practical use and the necessity of the claim normalization task to combat
misinformation and to create a more informed society. Our contributions are as
follows:

  * •

We introduce the novel task of claim normalization, which seeks to pare down
the core claim outlined in a given text.

  * •

We present a meticulously curated high-quality dataset specifically tailored
for claim normalization of noisy social media posts.

  * •

We propose a robust framework for claim normalization, incorporating chain-of-
thought, in-context learning, and claim check-worthiness estimation, to
comprehend intricate claims.

  * •

To acquire more profound insights into the generated normalized claims, we
conduct a thorough human error analysis.

##  2 Related Work

#### Claim Analysis.

Over the past few decades, extensive research has been conducted to address
various claim-related issues. The existing body of research has primarily
focused on distinct aspects of claims, including claim detection Daxenberger
et al. (2017); Gupta et al. (2021); Sundriyal et al. (2021), claim check-
worthiness Barrón-Cedeno et al. (2018, 2020), claim span identification
Sundriyal et al. (2022), etc. By curating the AAWD corpus, Bender et al.
(2011) pioneered the efforts in claim detection, the foremost step in the
fact-checking tasks. Following this, linguistically motivated features,
including sentiment, syntax, context-free grammar, and parse trees, were
frequently used Daxenberger et al. (2017); Lippi and Torroni (2015); Levy et
al. (2017); Sundriyal et al. (2021). Recently, large language models (LLMs)
have also been used for claim detection Chakrabarty et al. (2019); Barrón-
Cedeno et al. (2020); Williams et al. (2020); Gupta et al. (2021). Claim
check-worthiness estimation stands as a vital research paradigm, encompassing
the evaluation of claims to determine their suitability for fact-checking. A
few studies attempted to develop end-to-end systems for mining check-worthy
statements Jaradat et al. (2018); Hassan et al. (2017); Konstantinovskiy et
al. (2021).

The majority of the existing studies on claims primarily concentrate on
adapting to texts that come from similar distributions or topics. Moreover,
they often rely on well-structured formal writing. In contrast, our objective
is to develop a system that specifically addresses the challenges posed by
social media claims and aims to generate the central claim in a more
simplified manner. To the best of our knowledge, none of the existing studies
tackled the intricate task of simplifying central claims found within long,
noisy social media posts.

#### Text Summarization.

The task of claim normalization is closely related to text summarization,
where given a lengthy document, the goal is to summarize it into a much
shorter document. Numerous works in text summarization have explored various
approaches utilizing large pre-trained seq2Seq models to generate high-quality
summaries Radford et al. (2019); Lewis et al. (2020a); Raffel et al. (2020).
Kryściński et al. (2019) introduced FactCC, a weakly-supervised BERT-based
entailment model, which augmented the dataset with artificially introduced
faithfulness errors. Durmus et al. (2020) proposed question-answering-based
evaluation frameworks that automatically generate questions from the source
article and the generated summary. By comparing the corresponding answers,
they aimed to evaluate the information consistency of the summaries. However,
these approaches primarily focus on general-purpose summarization and do not
provide means for models to generate summaries primarily focusing on specific
needs. To address this limitation, controlled summarization was introduced Fan
et al. (2018). One aspect of controlled summarization is length control, in
which users can set their preferred summary length Rush et al. (2015); Kikuchi
et al. (2016). Recent research has discovered that, despite their fluency and
coherence, state-of-the-art abstractive summarization systems produce
summaries with contradictory information.

While text summarization systems can assist in condensing social media posts
into shorter summaries, their primary goal is not to ensure verifiability. It
aims to capture the key points of the text rather than emphasizing the
specific claims within the text that need to be fact-checked. Claim
normalization, on the other hand, works on an entirely different level. It
necessitates a thorough understanding of the claims made in the text and
strives to ensure that the normalized claims are not only consistent with
factual information but also independently verifiable.

Despite the progress made in various aspects of claims and text summarization,
the task of claim normalization remains underexplored. In this work, we intend
to tackle this challenging problem by developing a robust approach
specifically tailored to the unique requirements of claims.

##  3 Dataset

Existing text summarization datasets have not specifically addressed the need
for claim-oriented summaries. To address this gap, we propose a novel dataset
CLAN (Claim Normalization), consisting of fact-checked social media posts
paired with concise claim-oriented summaries (known as normalized claim),
created by fact-checkers during the verification process. As a result, our
dataset is not subjected to external annotation, thus averting potential
biases and ensuring high-quality data.

###  3.1 Data Collection

We gathered our fact-checked post and claim pairs using two methods: (_i_)
Google Fact-Check Explorer333https://toolbox.google.com/factcheck/explorer and
(_ii_) ClaimReview Schema.444https://schema.org/ClaimReview

#### Google Fact-Check Explorer.

We acquired a list of fact-checks from multiple reputed fact-check sources
obtained via Google Fact-Check Explorer’s API (GFC). This data collection
pipeline followed a three-step process. First, we extracted the _title_ ,
which is usually a single-sentence short summary of the information being
fact-checked, and the _fact-check site URL_. This step yielded a total of
$22,405$ unique fact-checks. We then proceeded to retrieve the social media
post and the associated ‘claim review’ if they were available on the fact-
checking site. Due to the collected posts having already undergone fact-
checking and containing misleading claims, a significant number of them were
unavailable for inclusion in our dataset. Moreover, a significant number of
the posts only contained images or videos, which were unsuitable for our task
at hand. As a result, we were left with a considerably small number of
relevant instances. We also noted that in certain instances, the _title_ in
the Google Fact-Check Explorer and the _claim review_ were identical;
consequently, we included only one of them in the final dataset. As a result
of this meticulous procedure, each social media post obtained through this
process, had either one or two corresponding normalized claims.

|  Social Media Post |  Normalized Claim
---|---|---
$1$ |  Research into the dangers of cooking with aluminum foil has found that some of the toxic metal can contaminate food. Increased levels of aluminum in the body have been linked to osteoporosis, and Alzheimer’s disease. |  Cooking in Aluminum foil causes Alzheimer’s Disease.
$2$ |  Did you know when ur child turns 6. U can add them as authorized user to one of ur credit cards. Never give them card, & all payments u make from 6 to 18 goes to ur child credit too..ur kid will have a unbelievable credit score from years of payment history. |  6-year-old kids can be added as authorized users on all credit cards.
$3$ |  As if it couldn’t get any worse. #Hope4Cancer says #RootCanal causes #CANCER Solution… Rip Cancer patients teeth out. Monsters #FalseHope4Cancer #ProtectCancerPatients |  Having a root canal can cause cancer.
3-3 |  |  Root canal treatment causes cancer

Table 1: Examples of social media posts and their corresponding normalized
claims from CLAN. The first two examples come from the train set and each have
one reference normalized claim, while the last one comes from the test set,
having two reference normalized claims.

#### The ClaimReview Schema.

We targeted ClaimReview Schema elements with an entry for _reviewed items_ as
they were relevant to our requirements. Out of a total of $44,478$ entries,
only $22,428$ had this particular field. Therefore, we had to filter out the
remaining entries. Next, we extracted all the links to social media posts and
their corresponding claim reviews provided by the fact-checkers. As mentioned
before, we solely processed textual claims and we excluded other modalities,
such as audio or video. Further, we ensured that all the entries were in
English.

###  3.2 Data Statistics and Analysis

By using both of these data collection methods and exercising careful
consideration, we curated a total of $6,388$ instances. To ensure the creation
of a diverse and high-quality test set, we chose posts that comprised not one,
but two reference normalized claims (c.f. Sec 3.1). This enabled us to capture
different aspects and perspectives of the normalized claims by including
multiple references, thereby increasing the test set’s robustness and
reliability. The final dataset statistics are shown in Table 2, while
representative samples from our dataset are shown in Table 1.

Dataset | Train | Val | Test | Overall
---|---|---|---|---
Total no. of pairs | 5341 | 594 | 453 | 6388
Avg. length of posts | 39.52 | 37.12 | 57.97 | 44.87
Avg. length of claims | 16.47 | 17.24 | 15.41 | 16.37

Table 2: Statistics of our dataset CLAN.  Figure 2: Cosine similarity between
the social media posts and the normalized claims from our dataset CLAN.

We analyze the cosine similarities between the social media posts and the
corresponding normalized claims, as shown in Figure 2. The results reveal that
the cosine similarities are consistently low for most examples, demonstrating
that claim normalization involves more than just summarizing the social media
post. It requires a deeper understanding of the claim’s context, intent, and
specific details. It highlights the need for a specialized technique to
accurately identify, extract, and normalize the claims within social media
posts.

Figure 3: Illustration of our proposed approach, CACN prompt template employed
to generate normalized claims from the given social media post. The template
encompasses explicit task instruction and relevant in-context examples.

##  4 Proposed Approach

In this section, we endeavor to explicate the methodology underpinning our
proposed approach, Check-worthiness Aware Claim Normalization (CACN). CACN
aims to integrate indispensable task-specific information with LLMs, elevating
their inherent reasoning prowess to unprecedented heights. We employ GPT-3
(text-davinci-003) Brown et al. (2020) as our foundation LLM. Our approach
amalgamates two key ideas – the paradigm of chain-of-thought prompting and the
concept of reverse check-worthiness.

#### Chain-of-thought Prompting.

The realm of chain-of-thought prompting emerges as a veritable tour de force
within LLMs Wei et al. (2022). Instead of undergoing the laborious process of
fine-tuning individual model checkpoints for every new task, we adopt a more
recent and efficient approach. We strategically prompt the model to navigate
the complexities of claim normalization by employing step-by-step reasoning.
To accomplish this, we utilize the concept of claim check-worthiness, as
described in the following subsection. This enables the model to iteratively
enhance its comprehension and effectively generate precise normalized claims,
all while minimizing the necessity for extensive fine-tuning. Our proposed
prompt example is shown in Figure 3. The chain-of-thought strategy creates a
paradigm shift when approaching a complicated problem by efficiently breaking
it into intermediate stages.

#### Reverse Check-Worthiness.

The premise of reverse check-worthiness originates from the check-worthiness
estimation, a facet prominently harnessed by manual fact-checkers Nakov et al.
(2021). We leverage this to steer the model’s attention toward salient and
pertinent information. By giving the model the ability to produce rationales
in natural language that clearly explain the sequence of reasoning stages
leading to the solution, we strengthen its capacity for cognitive reasoning
with unwavering efficacy. Based on prior research on claim check-worthiness
Barrón-Cedeno et al. (2018); Shaar et al. (2021); Nakov et al. (2022), we
direct our model to prioritize claims that meet specific criteria within the
given social media post. These criteria include identifying claims within
social media posts that contain verifiable factual content, have a higher
likelihood of being false, relevant to the general public’s interest, and are
worth fact-checking. For instance, in Figure 3, the claim normalization
process begins by identifying the central claim within the post. Subsequently,
we reckon the claim’s verifiability, determining whether it is sufficient to
support or refute it. We further evaluate the likelihood of the claim being
false and its overall check-worthiness. This step-by-step approach ensures a
comprehensive analysis of the central claim’s characteristics, allowing for
effective claim normalization. By incorporating these factors into our
approach, we aim to improve the model’s ability to identify and prioritize
claims that require scrutiny and verification.

##  5 Experimental Setup

#### Baseline Models.

For comparison, we employ several state-of-the-art generative systems and
categorize them into two groups – (i) Pre-trained Large Language Models
(PLMs): T5 Raffel et al. (2020), BART Lewis et al. (2020b), FLAN-T5 Chung et
al. (2022), and PEGASUS Zhang et al. (2020). For T5, BART, and FLAN-T5, we use
base and large model sizes. For PEGASUS, we employ reddit model. (ii) In-
context Learning Model: GPT-3 (text-davinci-003) Brown et al. (2020).

#### Evaluation Measures.

To evaluate lexical overlap, we utilize ROUGE ($1,2,L$) and BLEU-4 Papineni et
al. (2002). Moreover, we use METEOR Banerjee and Lavie (2005) and BERTScore
Zhang et al. (2019), to assess semantic similarity between the gold and the
generated normalized claims.

#### Zero-shot Learning.

In recent years, there has been a surge of interest in zero-shot learning,
which aims to apply the previously acquired capabilities of PLMs to similar
tasks in a low domain. We hereby study the zero-shot performance of these PLMs
to assess their suitability for the claim normalization task.

#### Few-shot Learning.

We adopt few-shot learning with $10$, $20$, $50$, and $100$ training examples.
This gradual exposure to additional labeled data aims to enhance the models’
ability to generate accurate and contextually appropriate normalized claims.

#### Prompt Tuning.

Prompt-tuning entails adding a specific prefix to the model’s input customized
to the downstream tasks Zhang et al. (2022). We investigate the effect of
affixing different prompts to the given posts on the performance of T5-based
and GPT-3 models. To exert control over the generated normalized claims, we
use five control aspects: tokens, abstractness, sentence count, claim-centric,
and entity-centric. A comprehensive description of these prompts is given in
the Appendix (A.2).

#### In-context Learning.

LLMs possess the remarkable ability to tackle diverse tasks with a minimal
amount of examples given in prompts Brown et al. (2020). We use GPT-3 (text-
davinci-003)’s for in-context learning, with three different prompts:direct
prompt (DIRECT), question-guided (Q-GUIDED), and zero-shot chain-of-thought
(ZS-CoT). We perform in-context learning with GPT-3 (text-davinci-003).
Detailed prompt templates are given in Appendix A.3.

| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4 | METEOR | BERTScore
---|---|---|---|---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1
Finetune | T5BASE | 23.65 | 43.60 | 28.99 | 11.38 | 20.70 | 13.90 | 20.45 | 37.91 | 25.15 | 4.57 | 28.86 | 85.13
T5LARGE | 23.37 | 44.81 | 28.99 | 10.98 | 21.11 | 13.66 | 20.07 | 38.46 | 24.97 | 4.43 | 29.33 | 85.09
BARTBASE | 33.41 | 41.64 | 34.11 | 17.57 | 21.05 | 17.55 | 29.70 | 36.61 | 30.25 | 6.69 | 29.57 | 86.32
BARTLARGE | 35.83 | 42.88 | 36.12 | 19.25 | 21.73 | 18.97 | 31.64 | 37.65 | 31.93 | 7.71 | 31.07 | 86.92
FLAN-T5BASE | 22.50 | 47.38 | 28.79 | 10.65 | 22.28 | 13.61 | 19.15 | 40.14 | 24.50 | 4.44 | 29.07 | 84.66
FLAN-T5LARGE | 28.70 | 45.98 | 31.35 | 15.32 | 22.63 | 16.24 | 25.46 | 39.89 | 27.63 | 6.62 | 29.36 | 84.65
In-context | DIRECT | 32.19 | 46.43 | 35.52 | 14.19 | 20.60 | 15.65 | 27.32 | 39.75 | 30.31 | 6.52 | 33.25 | 88.87
Q-GUIDED | 33.48 | 44.50 | 35.40 | 15.34 | 20.42 | 16.15 | 29.10 | 38.88 | 30.90 | 6.56 | 32.13 | 88.81
ZS-CoT | 27.77 | 48.39 | 32.42 | 13.18 | 22.33 | 15.16 | 23.84 | 41.51 | 27.79 | 6.37 | 32.93 | 88.44
CACN (ours) | 37.54 | 46.10 | 38.64 | 18.85 | 23.08 | 19.32 | 33.14 | 40.92 | 34.30 | 9.66 | 35.10 | 89.00
| $\Delta_{\texttt{CACN}-BEST(\%)}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 4.77}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow 4.73}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 6.98}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow 2.08}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 1.99}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 1.85}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 4.74}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow 1.42}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 7.42}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 25.29}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 5.56}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow 1.15}$

Table 3: Experimental results of CACN and baseline systems on CLAN. We report
ROUGE ($1,2,L$), BLEU-4, METEOR, and BERTScore. Highest scores are shown in
bold, while the second-highest scores are underlined, across each metric. The
last row gives the percentage increase in performance between CACN and the
best-performing baseline.

N-shot | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4 | METEOR | BERTScore
---|---|---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
0 | T5BASE | 23.38 | 42.25 | 27.79 | 11.16 | 19.54 | 13.14 | 20.29 | 36.52 | 24.10 | 4.30 | 28.45 | 85.36
T5LARGE | 24.80 | 43.71 | 29.08 | 12.44 | 20.93 | 14.31 | 21.58 | 37.82 | 25.30 | 5.01 | 29.36 | 85.65
BARTBASE | 22.57 | 47.76 | 27.96 | 11.24 | 22.60 | 13.60 | 19.58 | 40.91 | 24.14 | 4.64 | 30.44 | 85.11
BARTLARGE | 20.80 | 39.68 | 24.41 | 9.86 | 17.20 | 11.16 | 18.23 | 33.85 | 21.20 | 3.92 | 24.67 | 84.78
FLAN-T5BASE | 30.78 | 32.33 | 28.73 | 15.03 | 15.17 | 13.78 | 28.08 | 28.99 | 26.02 | 4.76 | 23.27 | 83.80
FLAN-T5LARGE | 31.20 | 34.06 | 30.22 | 15.67 | 16.83 | 15.07 | 28.41 | 30.57 | 27.40 | 5.76 | 25.82 | 84.89
PEGASUS | 23.96 | 36.07 | 26.16 | 11.69 | 16.76 | 12.48 | 21.14 | 31.50 | 22.99 | 4.87 | 24.87 | 83.07
10 | T5BASE | 21.55 | 44.22 | 27.44 | 9.87 | 20.36 | 12.57 | 18.31 | 37.77 | 23.40 | 4.04 | 28.64 | 84.97
T5LARGE | 21.92 | 45.73 | 28.04 | 10.17 | 21.04 | 12.98 | 18.64 | 38.81 | 23.85 | 4.19 | 29.52 | 85.11
BARTBASE | 20.09 | 53.08 | 25.65 | 10.14 | 24.94 | 12.60 | 17.34 | 44.77 | 22.00 | 4.11 | 29.67 | 84.72
BARTLARGE | 19.71 | 53.22 | 25.30 | 9.83 | 24.87 | 12.27 | 17.01 | 45.22 | 21.70 | 4.00 | 29.51 | 84.65
FLAN-T5BASE | 23.05 | 42.60 | 27.97 | 10.58 | 19.48 | 12.84 | 19.83 | 36.58 | 24.14 | 4.39 | 28.04 | 85.34
FLAN-T5LARGE | 22.41 | 29.66 | 19.99 | 10.68 | 13.18 | 9.49 | 20.50 | 25.91 | 17.84 | 3.04 | 18.87 | 78.04
PEGASUS | 15.26 | 40.32 | 20.86 | 7.15 | 18.29 | 9.58 | 13.39 | 35.87 | 18.35 | 3.44 | 24.73 | 80.02
20 | T5BASE | 21.56 | 44.22 | 27.46 | 9.89 | 20.36 | 12.59 | 18.33 | 37.78 | 23.42 | 4.05 | 28.64 | 84.97
T5LARGE | 21.97 | 45.80 | 28.07 | 10.24 | 21.11 | 13.03 | 18.69 | 38.77 | 23.86 | 4.21 | 29.46 | 85.12
BARTBASE | 20.08 | 53.16 | 25.63 | 10.14 | 25.03 | 12.60 | 17.34 | 44.89 | 21.99 | 4.11 | 29.63 | 84.72
BARTLARGE | 19.66 | 53.11 | 25.25 | 9.80 | 24.84 | 12.24 | 16.96 | 45.08 | 21.64 | 3.97 | 29.46 | 82.69
FLAN-T5BASE | 23.15 | 42.80 | 28.11 | 10.63 | 19.58 | 12.91 | 19.93 | 36.76 | 24.27 | 4.44 | 28.20 | 85.34
FLAN-T5LARGE | 22.38 | 29.79 | 20.00 | 10.69 | 13.25 | 9.51 | 20.48 | 26.00 | 17.85 | 3.04 | 18.96 | 78.01
PEGASUS | 15.25 | 40.30 | 20.84 | 7.15 | 18.29 | 9.58 | 13.38 | 35.87 | 18.35 | 3.44 | 24.72 | 80.01

Table 4: Zero-shot and few-shot results on our dataset CLAN. We show ROUGE
($1,2,L$), BLEU-4, METEOR, and BERTScore, to assess the performance of the
models.

##  6 Performance Comparison

Our findings reveal that our CACN surpasses all the baseline systems across
most evaluation measures. We thoroughly examine all the systems using the
following probes.

#### Do meticulously crafted prompts enhance the performance of generative
models?

The findings exhibit a significant performance improvement when prompt tuning
is used, specifically with in-context examples. Table 3 shows the
effectiveness of various prompts across all evaluation measures. However, a
notable enhancement of approximately $2$-$3$% is observed in all semantic
measures when transitioning from conventional prompts to our proposed
approach, with the same in-context examples. This emphasizes the importance of
our meticulously designed framework tailored to the specific task. Moreover,
an upsurge in ROUGE-F1 scores ($1,2,L$) emphasizes the resemblance between the
generated normalized claims and those created by humans. This, in turn,
validates the incorporation of the “ _reverse check-worthiness_ ” chain-of-
thought process, which effectively integrates task-specific information into
the generative system. We also attempt prompt-tuning in a zero-shot setup; the
results are shown in the Appendix (A.2). To summarize, the deliberate design
of prompts, along with in-context examples, substantially enhances the
performance of generative models, striking a promising equilibrium between
lexical and semantic grounds.

#### Is training models on a specific task less effective than in-context
learning with a few examples?

We observe substantial disparities in the performance of models trained on
task-specific data compared to using in-context learning with a limited number
of examples, as shown in Table 3. The models exposed to in-context examples
showcase superior performance, highlighting their efficacy in capturing task-
specific patterns. While the trained models exhibit excellence in lexical
metrics, their performance in semantic metrics is noticeably lower. Notably,
BARTLARGE, trained on our dataset, outperforms other trained models by huge
margins. These results strongly underline that, within the realm of LLMs,
incorporating prompt-tuning with in-context learning holds more promise,
leading to enhanced generalization capabilities of these models.

#### Do models demonstrate inherent proficiency in generating normalized
claims with minimal or no prior training?

We examine the potential benefits of zero-shot and few-shot learning to
investigate the inherent proficiency in generating normalized claims. The
zero-shot and the few-shot results are shown in Table 4. Zero-shot learning,
which relies solely on the pre-trained language model without any task-
specific fine-tuning, performs quite well. On the other hand, few-shot
learning does not result in significant improvements in model performance.
Surprisingly, the models trained using few-shot learning perform slightly
worse than the zero-shot approach, in which the models have no exposure to
task-specific data. After training on ten samples, the performance of
FLAN-T5LARGE drops by $6$ BERTScore points, and it continues to decline as
more examples are provided.555See Appendix (A.4) for $50$ and $100$-shot
results. This unexpected result suggests that few-shot learning may be
unsuitable for this intricate and complex task. The limited number of examples
provided during few-shot learning may have been insufficient for the models to
effectively generalize and capture the underlying patterns of normalized
claims. Additionally, the introduction of task-specific data might have
introduced conflicting information as these models were never trained on this
task, leading to a degradation in performance.

##  7 Qualitative Analysis

| Social Media Post |  Normalized Claim | BS
---|---|---|---
Sample 1 | They gave you a MASK to cover your face AND then gave you George Floyd’s signature phrase ‘I can’t breath’ …. Yet 2.5 years later, most still struggle with the realization that EVERYTHING on TV is fake. Floyd and ConVid never existed |  GOLD: George Floyd and COVID-19 ‘never existed’. | –
BART: George Floyd died of COVID-19 | 90.33
DIRECT: George Floyd’s death highlighted the lack of understanding of the reality of events portrayed on television. | 86.84
CACN: George Floyd and Covid never existed. | 90.98
Sample 2 | Good News. Finally, CAJY VAZ, an Indian EX Student from St. Xavier’s High School from Mumbai, presently based in Goa, found a home remedy for *Covid 19*, which was approved by the WOH for the first time. He proved that *one teaspoon of pepper powder, two teaspoons of honey, a little ginger juice taken for 5 consecutive days can eliminate the effect of corona up to 100%*. The whole world is starting to take this precautionary treatment, finally a happy experience of 2021. |  GOLD: Honey, Ginger, Black Pepper can cure Coronavirus. | –
BART: One teaspoon of pepper powder, two teaspoons of honey, a little ginger juice taken for 5 consecutive days can eliminate the effect of Covid up to 100%. | 84.89
DIRECT: Indian ex-student Cajy Vaz found a home remedy of pepper powder, honey and ginger juice that can reduce the effects of Covid-19 up to 100%. | 84.65
CACN: A home remedy of pepper powder, honey, and ginger juice can eliminate the effect of COVID-19 up to 100%. | 78.47
Sample 3 | Zelensky sold 17 million hectares of agricultural land to Monsanto, Dupont, and Cargill. Yes, you read it well, 17 million hectares to GM0/chemical companies. This is very bad for the entire world since Ukraine is the largest exporter of wheat and other grains. Zelensky sold 28% of the entire Ukrainian arable land. Australian National Review reports three major US cross-border consortiums have bought 17 million hectares of Ukrainian farmland. To compare: In all of Italy, there are 16.7 million hectares of agricultural land. It turns out that three American companies in Ukraine bought more valuable agricultural land than in all of Italy. The entire area of Ukraine - 600,000 sqm, 170,000 sqm built. |  GOLD: Ukraine President Volodymyr Zelenskyy sold 17 million hectares of farmland to US corporations. | –
BART: Three American companies have bought 17 million hectares of farmland in Ukraine. | 88.03
DIRECT: Three major US companies have bought 17 million hectares of Ukrainian agricultural land, which is more than in all of Italy. These companies are Cargill, Dupont and Monsanto, with their main shareholders being American venture capitalists Blackrock, Vanguard and Blackstone. | 83.57
CACN: Ukrainian President Zelensky sold 17 million hectares of agricultural land to Monsanto, Dupont, and Cargill. | 90.48

Table 5: Examples showing the generated normalized claims along with the gold
reference from our test set. BART refers to BARTLARGE.

#### Error Analysis.

To comprehend the performance of CACN, we strive to qualitatively analyze the
errors committed by our model in this section. Table 5 displays randomly
selected instances from our test dataset, along with gold normalized claims
and predictions from CACN. We also furnish predictions from two best-
performing baselines, BARTLARGE and DIRECT, for comparison. The predictions in
the fine-grained analysis are, naturally, much more intricate than in the
coarse-grained quantitative setup. During our analysis, we unveil several
interesting patterns and errors in the generated responses. Although the
BARTLARGE generated responses have a high BERTScore in Sample $1$, we notice
that the factual alignment is incorrect, making it untrustworthy for
downstream tasks such as claim check-worthiness and claim verification. Our
model, on the other hand, produces a response that is both correct and
precise. The response generated by DIRECT is also accurate but excessively
long, which contradicts the goal of normalized claims being concise and
straightforward. This problem is also evident in Sample $3$, where DIRECT
produces factually correct information but is overly long. In Sample $2$, we
observe that BARTLARGE demonstrates the least amount of hallucinations and
adheres closely to the given post. Our model’s BERTScore is the lowest in this
case. However, upon closer inspection, we determine that the generated
normalized claim is indeed correct and most relevant for fact-checking. These
findings highlight the complexities and the trade-offs involved in generating
normalized claims. While certain models may excel in certain aspects, there is
often a compromise in other areas, such as factual accuracy and conciseness.

Model | Fluency | Coherence | Relevance | Consistency | SC
---|---|---|---|---|---
BART | 3.44 | 3.74 | 3.66 | 3.82 | 3.77
DIRECT | 4.48 | 4.58 | 4.03 | 4.26 | 4.38
CACN | 4.59 | 4.63 | 4.17 | 4.34 | 4.39

Table 6: Human evaluation on the generated normalized claims. SC denotes Self-
Contextualized, while BART refers to BARTLARGE .

#### Human Evaluation.

We conducted an extensive human evaluation to assess the linguistic
proficiency of the generated normalized claims. Building upon the measures
proposed by van der Lee et al. (2021), we evaluated the generated claims based
on four aspects: fluency, coherence, relevance, and factual consistency.666See
Appendix A.6 for more detail. Additionally, we introduced the parameter of
self-contextualized to measure the extent to which the normalized claims
included the necessary context for fact-checking within themselves. Each of
these measures played a unique and vital role in evaluating the quality of the
generated claims. To conduct the evaluation, we randomly selected $50$
instances from our test set and assigned five human evaluators to rate every
normalized claim on a scale of $1$ to $5$ for each of the five aspects. All
evaluators were fluent English speakers with a BSc or MSc degree. To ensure
reliability, every example was evaluated by all five evaluators independently.
For comparison, we also included the results from the best-performing baseline
systems, namely BARTLARGE and DIRECT. The average scores are presented in
Table 6. Our analysis reveals that the outputs generated by CACN exhibit
qualitative superiority compared to the baseline systems across all
dimensions.

##  8 Conclusion

We presented a new problem formulation and a new dataset, thus contributing to
research on claim extraction and, ultimately, fact-checking. We introduced the
novel task of claim normalization, which holds substantial value on multiple
fronts. For human fact-checkers, claim normalization is a useful tool that can
assist them in effectively removing superfluous texts from subsequent
processing. This also benefits downstream tasks such as identifying previously
fact-checked claims, estimating claim check-worthiness, etc. We compiled a
social media corpus comprising over $6k$ posts and their normalized claims. We
further benchmarked this dataset with a novel approach, CACN, and showed its
superior performance compared to different state-of-the-art generative models
across multiple assessment measures. We also documented our extensive data
collection process, providing valuable insights for future research in this
domain.

In future work, we plan to extend the dataset, including with new languages.
We also plan to use more powerful LLMs.

## Limitations

While our study has made major contributions to claim normalization, it is
critical to recognize and address the potential limits associated with it.
During our data collection process, we observed that several fact-checked
claims also contained images and videos. As a result, we believe that
including multimodal information may aid in improving the claim normalization
process. Another key problem stems from the subjectivity of fact-checkers
writing styles across different fact-checking websites. Each organization
adheres to its own set of editorial norms, procedures, and subjective
interpretations of claims. These variations in writing styles and subjective
judgments make it challenging to establish a standardized approach. Addressing
this issue will necessitate attempts to develop consensus or guidelines among
fact-checking organizations in order to ensure greater consistency and
coherence in claim normalization processes. By acknowledging and addressing
these limitations, we may endeavor to improve the reliability and soundness of
claim normalization systems in the future.

## Ethics and Broader Impact

#### Data Bias.

It is imperative to acknowledge the possibility of biases within our dataset.
Our data collection process involves gathering normalized claims from multiple
fact-checking sites, each with its own set of editorial norms, procedures, and
subjective interpretations. These elements can introduce systemic biases that
impact the overall assessment of normalized claims. However, it is important
to acknowledge that these biases are beyond our control.

#### Environmental Footprint.

Large language models (LLMs) require a substantial amount of energy for
computational training, which adds to global warming Strubell et al. (2019).
Our proposed approach, on the other hand, leverages few-shot in-context
learning rather than training models from scratch, leading to a lower carbon
footprint. It is worth mentioning that the LLMs usage for inference still
costs a considerable amount of energy, and we are actively seeking to reduce
it by employing more energy-efficient techniques.

#### Broader Impact and Potential Use.

Our model can interest the general public and save time for human fact-
checkers. Its applications extend beyond fact-checking to other downstream
tasks like detecting previously fact-checked claims, claim matching, and even
estimating claim check-worthiness of new claims.

## References

  * Allcott and Gentzkow (2017) Hunt Allcott and Matthew Gentzkow. 2017.  Social media and fake news in the 2016 election.  _Journal of Economic Perspectives_ , 31(2):211–36.
  * Angeli et al. (2015) Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015.  Leveraging linguistic structure for open domain information extraction.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 344–354.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Barrón-Cedeno et al. (2020) Alberto Barrón-Cedeno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, and Fatima Haouari. 2020.  Checkthat! at clef 2020: Enabling the automatic identification and verification of claims in social media.  In _European Conference on Information Retrieval_ , pages 499–507. Springer, Nature Publishing Group.
  * Barrón-Cedeno et al. (2018) Alberto Barrón-Cedeno, Tamer Elsayed, Reem Suwaileh, Lluís Màrquez, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018.  Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. task 2: Factuality.  _CLEF (Working Notes)_ , 2125.
  * Bender et al. (2011) Emily M. Bender, Jonathan T. Morgan, Meghan Oxley, Mark Zachry, Brian Hutchinson, Alex Marin, Bin Zhang, and Mari Ostendorf. 2011.  Annotating social acts: Authority claims and alignment moves in Wikipedia talk pages.  In _Proceedings of the Workshop on Language in Social Media (LSM 2011)_ , pages 48–57, Portland, Oregon. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chakrabarty et al. (2019) Tuhin Chakrabarty, Christopher Hidey, and Kathleen McKeown. 2019.  Imho fine-tuning improves claim detection.  _arXiv preprint arXiv:1905.07000_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Daxenberger et al. (2017) Johannes Daxenberger, Steffen Eger, Ivan Habernal, Christian Stab, and Iryna Gurevych. 2017.  What is the essence of a claim? cross-domain claim identification.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2055–2066, Copenhagen, Denmark. Association for Computational Linguistics.
  * Dreyer et al. (2023) Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2023.  Evaluating the tradeoff between abstractiveness and factuality in abstractive summarization.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 2044–2060.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018.  Controllable abstractive summarization.  _ACL 2018_ , page 45.
  * Gupta et al. (2021) Shreya Gupta, Parantak Singh, Megha Sundriyal, Md. Shad Akhtar, and Tanmoy Chakraborty. 2021.  LESA: Linguistic encapsulation and semantic amalgamation based generalised claim detection from online content.  In _Proc of EACL_ , pages 3178–3188, Online. ACL.
  * Hanselowski et al. (2018) Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018.  UKP-athene: Multi-sentence textual entailment for claim verification.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , pages 103–108, Brussels, Belgium. Association for Computational Linguistics.
  * Hassan et al. (2017) Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al. 2017.  Claimbuster: The first-ever end-to-end fact-checking system.  _Proc. of the VLDB Endowment 2017_ , 10(12):1945–1948.
  * Jaradat et al. (2018) Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedeño, Lluís Màrquez, and Preslav Nakov. 2018.  ClaimRank: Detecting check-worthy claims in Arabic and English.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_ , pages 26–30, New Orleans, Louisiana. Association for Computational Linguistics.
  * Khaldarova and Pantti (2020) Irina Khaldarova and Mervi Pantti. 2020.  Fake news: The narrative battle over the ukrainian conflict.  In _The Future of Journalism: Risks, Threats and Opportunities_ , pages 228–238. Routledge.
  * Kikuchi et al. (2016) Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016.  Controlling output length in neural encoder-decoders.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1328–1338.
  * Konstantinovskiy et al. (2021) Lev Konstantinovskiy, Oliver Price, Mevan Babakar, and Arkaitz Zubiaga. 2021.  Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection.  _Digital threats: research and practice_ , 2(2):1–16.
  * Kryściński et al. (2019) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019\.  Evaluating the factual consistency of abstractive text summarization.  _arXiv preprint arXiv:1910.12840_.
  * Levy et al. (2017) Ran Levy, Shai Gretz, Benjamin Sznajder, Shay Hummel, Ranit Aharonov, and Noam Slonim. 2017.  Unsupervised corpus–wide claim detection.  In _Proceedings of the 4th Workshop on Argument Mining_ , pages 79–84.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proc. of ACL_ , pages 7871–7880, Online. ACL.
  * Lewis et al. (2020b) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Lippi and Torroni (2015) Marco Lippi and Paolo Torroni. 2015.  Context-independent claim detection for argument mining.  In _Twenty-Fourth International Joint Conference on Artificial Intelligence_.
  * Nakov et al. (2022) Preslav Nakov, Alberto Barrón-Cedeño, Giovanni da San Martino, Firoj Alam, Julia Maria Struß, Thomas Mandl, Rubén Míguez, Tommaso Caselli, Mucahid Kutlu, Wajdi Zaghouani, et al. 2022.  Overview of the clef–2022 checkthat! lab on fighting the covid-19 infodemic and fake news detection.  In _CLEF (working notes)_ , pages 495–520. Springer.
  * Nakov et al. (2021) Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Watheq Mansour, et al. 2021.  Overview of the clef–2021 checkthat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news.  In _CLEF (Working Notes)_ , pages 264–291. Springer.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rocha et al. (2021) Yasmim Mendes Rocha, Gabriel Acácio de Moura, Gabriel Alves Desidério, Carlos Henrique de Oliveira, Francisco Dantas Lourenço, and Larissa Deadame de Figueiredo Nicolete. 2021.  The impact of fake news on social media and its influence on health during the covid-19 pandemic: A systematic review.  _Journal of Public Health_ , pages 1–10.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  _arXiv preprint arXiv:1509.00685_.
  * Shaar et al. (2021) Shaden Shaar, Maram Hasanain, Bayan Hamdan, Zien Sheikh Ali, Fatima Haouari, Alex Nikolov, Mücahid Kutlu, Yavuz Selim Kartal, Firoj Alam, Giovanni Da San Martino, et al. 2021.  Overview of the clef-2021 checkthat! lab task 1 on check-worthiness estimation in tweets and political debates.  In _CLEF (Working Notes)_.
  * Soleimani et al. (2020) Amir Soleimani, Christof Monz, and Marcel Worring. 2020.  Bert for evidence retrieval and claim verification.  _Advances in Information Retrieval_ , 12036:359.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in nlp.  _ArXiv_ , abs/1906.02243.
  * Sundriyal et al. (2022) Megha Sundriyal, Atharva Kulkarni, Vaibhav Pulastya, Md Shad Akhtar, and Tanmoy Chakraborty. 2022.  Empowering the fact-checkers! automatic identification of claim spans on twitter.  _arXiv preprint arXiv:2210.04710_.
  * Sundriyal et al. (2021) Megha Sundriyal, Parantak Singh, Md Shad Akhtar, Shubhashis Sengupta, and Tanmoy Chakraborty. 2021.  Desyr: Definition and syntactic representation based claim detection on the web.  _arXiv preprint arXiv:2108.08759_.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _CoRR_ , abs/2201.11903.
  * Williams et al. (2020) Evan Williams, Paul Rodrigues, and Valerie Novak. 2020.  Accenture at checkthat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models.  _arXiv preprint arXiv:2009.02431_.
  * Wright and Augenstein (2020) Dustin Wright and Isabelle Augenstein. 2020.  Claim check-worthiness detection as positive unlabelled learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 476–488, Online. Association for Computational Linguistics.
  * Zhang et al. (2022) Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022.  A survey of controllable text generation using transformer-based pre-trained language models.  _arXiv preprint arXiv:2201.05337_.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhi et al. (2017) Shi Zhi, Yicheng Sun, Jiayi Liu, Chao Zhang, and Jiawei Han. 2017.  Claimverif: A real-time claim verification system using the web and fact databases.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , CIKM ’17, page 2555–2558, New York, NY, USA. Association for Computing Machinery.

##  Appendix A Appendix

###  A.1 Task Motivation

Claim normalization holds significant promise in combating the spread of
misinformation by streamlining fact-checking processes and enhancing the
reliability of retrieved evidence. To substantiate our hypothesis regarding
the effectiveness of claim normalization, we conducted a well-structured
retrieval experiment using the Google API. The objective was to demonstrate
the practical benefits of claim normalization in assisting fact-checkers. We
randomly selected a sample of $35$ instances from our dataset, encompassing
social media posts and their normalized claims. Leveraging the capabilities of
the Google API, we sought the top $5$ most relevant articles for each post and
its normalized claim. In a meticulous evaluation process, three annotators
individually assessed the relevance ($0$ or $1$) of each retrieved article to
the input (post or normalized claim). We then employed majority voting to
determine the final relevance score for each retrieved article.

As depicted in Table 7, the results of our experiment consistently
demonstrated the advantage of normalized claims in evidence retrieval. In
top-$k$ precision evaluations for various values of $k$ $(1,3,$ and $5)$,
normalized claims consistently outperformed their corresponding source posts.
This observation indicates that claim normalization is not merely a
theoretical concept; but significantly enhances the efficiency of evidence
retrieval, resulting in more concise and effective tools for aiding the fact-
checking process.

| P@$1$ | P@$3$ | P@$5$
---|---|---|---
Post | 0.82 | 0.64 | 0.58
Normalized Claim | 0.88 | 0.73 | 0.69

Table 7: Comparative top-$k$ precision evaluations of Normalized Claim vs.
Original Posts in Evidence Retrieval

###  A.2 Prompt-tuning

Raffel et al. (2020) demonstrated that prompt-tuning could enable controllable
text generation in T5-based models. We also investigate the effect of affixing
different prompts to the given input on the performance of T5-based models for
normalized claim generation, along with GPT-3 (text-davinci-003) Brown et al.
(2020). We concoct various prompts that are suffixed with the input text
before inference, in a zero-shot setting. We discuss our different prompts
$P_{i}$ as follows:

Uncontrolled. We investigate the use of the traditional prompt ‘summarize’ for
uncontrollable models. This prompt lacks specific control signals, making it
uncontrolled as it does not provide explicit guidance with specific
attributes.

Token Limit. In our analysis, we found that normalized claims written by fact-
checkers typically adhere to around ten words, as shown in Figure 4. Thus, to
control the length of normalized claims, we utilize the prompt ‘summarize
within the length of 10 tokens’.

Figure 4: Box-plot for the number of tokens in normalized claims in CLAN.

Abstractness. Abstractness quantifies how much the generated text’s words, and
phrases differ from those extracted directly from it. Precisely, the more
n-grams overlap between a summary and its original document, the less
abstractive a summary is. Thus, to control the abstractness of the generated
normalized claims, we utilize the prompt ‘summarize with abstractness of $a$’
where ‘a’ represents a value within the range of 0 to 1, denoting the desired
level of abstractness. Inspired by Dreyer et al. (2023), we compute
abstractness score ($a_{i}$) for each pair of a post ($p_{i}$)) and a
normalized claim ($s_{i}$), and then we take an average across all examples:

$a_{i}=(1-X(p_{i},s_{i}))$ (1)

where we define $X$ as the harmonic mean of unigram overlaps precision, bi-
gram overlap precision, and longest sub-sequence overlap precision. We found
the average abstractness ($a$) to be around $0.8$.

Single Sentence. The normalized claim written by fact-checkers often consists
of a concise one-sentence summary of the post. We use the prompt ‘summarise in
one sentence’ to limit the normalized claims to a single-sentence summary,
ensuring brevity and conciseness in delivering the pivotal assertion.

Claim-centric. The task at hand is more than just a text summary; it
transcends the conventional approach by condensing the text and seeking to
discern and to encapsulate the central claim within the text concisely. Thus,
we formulate the prompt ‘summarize the text identifying the central assertion’
to helm the model to focus on the central assertion in the given text.

Entity-centric. Similarly to the claim-centric prompt, we investigate the
technique of creating entity-centered summaries using the prompt, ‘summarize
the text focusing on the given keywords $({kw_{1},kw_{2},kw_{3},...kw_{n}})$’.
In this approach, we use Open Information Extraction (OpenIE) Angeli et al.
(2015) to extract subject-verb-object triples from the input
texts.777https://github.com/philipperemy/Stanford-OpenIE-Python Subsequently,
we compile a keyword list encompassing all subjects and all objects within the
text. The objective is to direct the model in producing summaries that align
most with the subject and the object mentioned in the text.

| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4 | METEOR | BERTScore
---|---|---|---|---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1
T5BASE | UNCONTROLLED | 23.38 | 42.25 | 27.79 | 11.16 | 19.54 | 13.14 | 20.29 | 36.52 | 24.10 | 4.30 | 28.45 | 85.36
TOKEN-LIMIT | 22.87 | 42.71 | 27.59 | 11.42 | 20.61 | 13.60 | 20.21 | 37.47 | 24.34 | 4.60 | 28.67 | 85.31
ABSTRACTNESS | 23.02 | 42.21 | 27.58 | 11.40 | 20.20 | 13.47 | 20.17 | 36.79 | 24.16 | 4.53 | 28.46 | 85.41
SINGLE SENTENCE | 21.06 | 31.19 | 23.07 | 9.61 | 14.02 | 10.46 | 18.30 | 26.75 | 19.96 | 3.57 | 21.77 | 84.41
CLAIM-CENTRIC | 19.76 | 38.89 | 24.20 | 8.90 | 17.94 | 10.96 | 17.37 | 34.33 | 21.38 | 3.51 | 25.77 | 84.58
ENTITY-CENTRIC | 19.08 | 32.88 | 22.63 | 8.37 | 14.29 | 9.91 | 16.66 | 28.70 | 19.81 | 3.26 | 20.11 | 83.20
T5LARGE | UNCONTROLLED | 24.80 | 43.71 | 29.08 | 12.44 | 20.93 | 14.31 | 21.58 | 37.82 | 25.30 | 5.01 | 29.36 | 85.65
TOKEN-LIMIT | 14.51 | 36.24 | 19.53 | 6.24 | 16.01 | 8.45 | 12.79 | 32.47 | 17.33 | 2.56 | 22.13 | 82.08
ABSTRACTNESS | 23.53 | 44.44 | 28.62 | 11.69 | 20.98 | 13.96 | 20.66 | 38.64 | 25.10 | 5.02 | 29.74 | 85.58
SINGLE SENTENCE | 23.92 | 44.73 | 28.92 | 11.81 | 21.04 | 14.04 | 20.82 | 38.60 | 25.16 | 4.90 | 29.94 | 85.65
CLAIM-CENTRIC | 11.61 | 33.36 | 16.44 | 4.95 | 14.77 | 7.08 | 10.50 | 30.70 | 14.97 | 2.38 | 19.64 | 80.99
ENTITY-CENTRIC | 12.35 | 30.61 | 16.55 | 5.13 | 13.42 | 7.01 | 10.91 | 27.22 | 14.65 | 2.20 | 18.07 | 82.11
GPT-$3$ | UNCONTROLLED | 19.17 | 48.22 | 24.77 | 7.17 | 19.69 | 9.51 | 16.09 | 40.49 | 20.76 | 3.09 | 27.60 | 86.77
TOKEN-LIMIT | 26.29 | 40.66 | 28.92 | 9.75 | 15.03 | 10.59 | 22.24 | 34.07 | 24.34 | 3.75 | 26.25 | 87.11
ABSTRACTNESS | 15.74 | 54.44 | 23.27 | 6.14 | 22.29 | 9.17 | 12.52 | 44.37 | 18.63 | 2.84 | 29.50 | 86.91
SINGLE SENTENCE | 22.81 | 44.71 | 28.46 | 8.78 | 17.72 | 11.07 | 18.60 | 37.37 | 23.43 | 4.27 | 29.02 | 87.85
CLAIM-CENTRIC | 14.03 | 56.41 | 21.72 | 5.53 | 23.93 | 8.68 | 11.11 | 46.18 | 17.33 | 2.59 | 25.88 | 86.50
ENTITY-CENTRIC | 18.92 | 53.52 | 26.34 | 7.74 | 22.41 | 10.82 | 15.26 | 43.90 | 21.34 | 3.67 | 30.98 | 87.16

Table 8: Zero-shot prompt-tuning results for T5 and GPT-3 models on our
datatset CLAN.

#### Results.

We report the results of our zero-shot prompt-tuning in Table 8. We observed
that in T5-based models, prompt-tuning yielded no major improvements; rather,
it decreased the performance of the models as compared to the ‘uncontrolled’
prompt. However, GPT-3 (text-davinci-003) showed some improvements when using
these prompts.

###  A.3 In-Context Learning Templates

In Figure 5, we show the templates used for the three in-context learning
methods used for GPT-3 (text-davinci-003) as mentioned in Section 5.

###  A.4 Few-Shot Additional Results

We report 50 and 100-shot results in Table 9. We observe that introducing more
examples to the model, did not help much.

Training Samples | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-4 | METEOR | BERTScore
---|---|---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
50 | T5BASE | 21.57 | 44.24 | 27.46 | 9.88 | 20.36 | 12.58 | 18.32 | 37.78 | 23.41 | 4.05 | 28.64 | 84.97
T5LARGE | 22.00 | 45.86 | 28.11 | 10.24 | 21.11 | 13.03 | 18.71 | 38.83 | 23.90 | 4.22 | 29.49 | 85.11
BARTBASE | 20.10 | 53.16 | 25.65 | 10.15 | 25.02 | 12.61 | 17.35 | 44.93 | 22.00 | 4.11 | 29.66 | 84.71
BARTLARGE | 19.63 | 52.96 | 25.20 | 9.78 | 24.73 | 12.21 | 16.94 | 44.88 | 21.61 | 3.97 | 29.46 | 84.65
FLAN-T5BASE | 23.16 | 42.87 | 28.14 | 10.62 | 19.60 | 12.91 | 19.92 | 36.79 | 24.26 | 4.43 | 28.30 | 85.35
FLAN-T5LARGE | 22.38 | 29.63 | 19.97 | 10.69 | 13.25 | 9.51 | 20.49 | 25.92 | 17.84 | 3.03 | 18.92 | 77.84
PEGASUS | 15.24 | 40.28 | 20.83 | 7.15 | 18.29 | 9.58 | 13.38 | 35.84 | 18.34 | 3.44 | 24.71 | 80.02
100 | T5BASE | 21.55 | 44.24 | 27.45 | 9.87 | 20.36 | 12.57 | 18.31 | 37.78 | 23.40 | 4.04 | 28.65 | 84.97
T5LARGE | 21.97 | 45.72 | 28.06 | 10.24 | 21.06 | 13.02 | 18.69 | 38.78 | 23.86 | 4.22 | 29.47 | 85.11
BARTBASE | 20.09 | 53.28 | 25.65 | 10.15 | 24.97 | 12.60 | 17.35 | 44.91 | 22.00 | 4.11 | 29.61 | 84.72
BARTLARGE | 19.68 | 53.00 | 25.26 | 9.80 | 24.72 | 12.22 | 16.99 | 44.94 | 21.67 | 3.97 | 29.43 | 84.66
FLAN-T5BASE | 23.19 | 42.91 | 28.18 | 10.65 | 19.63 | 12.94 | 19.96 | 36.85 | 24.32 | 4.43 | 28.29 | 85.36
FLAN-T5LARGE | 22.38 | 29.63 | 19.97 | 10.69 | 13.25 | 9.51 | 20.49 | 25.92 | 17.84 | 3.03 | 18.92 | 77.85
PEGASUS | 15.30 | 40.30 | 20.88 | 7.16 | 18.27 | 9.58 | 13.41 | 35.89 | 18.37 | 3.43 | 24.74 | 80.02

Table 9: Few-shot results on our dataset CLAN.

###  A.5 Implementation Details

We perform basic data cleaning like removing non-alphanumeric characters,
removing links and hashtags, etc on our dataset CLAN, using nltk. For a
standardized evaluation, we rely on widely recognized evaluation libraries
such as py-rouge, nltk-bleu, nltk-meteor, and hugging-face bert-
score.888https://pypi.org/project/py-rouge/,
https://www.nltk.org/_modules/nltk/translate/bleu_score.html,
https://www.nltk.org/api/nltk.translate.meteor_score.html,
https://huggingface.co/spaces/evaluate-metric/bertscore We train all models
for 50 epochs, with early stopping based on validation loss. We set patience
value as 5 and optimize the models with Adam optimizer. Weight decay is set to
0.01. For our proposed approach CACN, we use GPT-3 (text-davinci-0003) as base
model. Max length of generated response is set to 120 with temperature 0.6.

###  A.6 Human Evaluation Criteria

Following van der Lee et al. (2021), we define the four human evaluation
metrics as follows:

  1. 1.

Fluency: Measures the linguistic proficiency exhibited by the generated
responses.

  2. 2.

Coherence: Evaluates the intrinsic structure and organization of the generated
normalized claims.

  3. 3.

Relevance: Appraises the discerning selection of contextually appropriate
content within the generated response.

  4. 4.

Factual Consistency: Examines the intricate alignment between the factual
accuracy of the generated response and the source text.

Figure 5: Templates for in-context learning prompts used for GPT-3 (text-
davinci-003).