# ChartCheck: An Evidence-Based Fact-Checking Dataset
over Real-World Chart Images

Mubashara Akhtar1, Nikesh Subedi 2, Vivek Gupta3, Sahar Tahmasebi4,
Oana Cocarascu1 and Elena Simperl1
1King’s College London 2University of Utah 3University of Pennsylvania
4TIB – Leibniz Information Centre for Science and Technology
<EMAIL_ADDRESS>
###### Abstract

Data visualizations are common in the real-world. We often use them in data
sources such as scientific documents, news articles, textbooks, and social
media to summarize key information in a visual form. Charts can also mislead
its audience by communicating false information or biasing them towards a
specific agenda. Verifying claims against charts is not a straightforward
process. It requires analyzing both the text and visual components of the
chart, considering characteristics such as colors, positions, and
orientations. Moreover, to determine if a claim is supported by the chart
content often requires different types of reasoning. To address this
challenge, we introduce ChartCheck, a novel dataset for fact-checking against
chart images. ChartCheck is the first large-scale dataset with $1.7k$ real-
world charts and $10.5k$ human-written claims and explanations. We evaluated
the dataset on state-of-the-art models and achieved an accuracy of $73.9$ in
the finetuned setting. Additionally, we identified chart characteristics and
reasoning types that challenge the models.

##  1 Introduction

Data visualizations like bar charts, pie charts and line graphs are common in
real-world data sources. We frequently use them in scientific documents, news
articles, textbooks, and social media to summarize key information in a visual
form. Information which is often not fully repeated in the associated text
(Carberry et al., 2006).

However, charts can also mislead to convey false messages, convince the
audience for a particular agenda, or spread misinformation (Lo et al., 2022).
For example, during the relatively recent COVID pandemic, the term “counter-
visualizations” was coined for charts circulating on social media platforms
with the aim of countering COVID-related health measurements (Lee et al.,
2021).

Verifying (misleading) statements against charts is not straightforward. To
understand charts, one needs to extract text and visual components (e.g.
lines, markers, or bars) from the image, while considering characteristics
like colors, position in the image, and orientation. Finally, different
reasoning types (e.g. retrieving values, filtering, calculation of sum or
average) are applied to classify claims veracity based on the chart.

Evidence Chart: Caption: Electricity production in Romania by source of energy. Based on data from ANRE $2010$. C$1$: Hydro is the primary source of energy for electricity production in Romania. E$1$: Hydro contributes to $36$% of the electricity production in Romania, which is the highest percentage among all the sources of energy listed in the chart. C$2$: Fifty percent of electricity in Romania in $2010$ came from gas as a power source. E$2$: Gas comprised ten percent of Romania’s electricity as a source of power in $2010$. |
---|---
Figure 1: A ChartCheck dataset sample. The chart evidence supports claim C$1$
and refutes C$2$. E$1$ and E$2$ are their explanations, respectively.

In order to address these challenges, we need a dataset that reflects the
variety of charts found in real-world sources. Additionally, the dataset
should include human-written claims that require the same type of chart
reasoning that humans employ to understand and analyse chart data. In this
paper, we introduce _ChartCheck_ , a novel dataset for fact-checking against
chart images. ChartCheck is the first large-scale fact-checking dataset with
$1.7k$ real-world charts and $10.5k$ human-written claims and explanations
(see Figure 1).

We evaluate state-of-the-art models for chart reasoning and explanation
generation in a fine-tuning, few- and zero-shot setting on ChartCheck. Hereby,
we consider two baseline architectures using state-of-the-art image-to-text
models (Lee et al., 2022a; Liu et al., 2022a, b): $(1)$ vision-language (VL)
models that use the chart image as direct input and $(2)$ baselines that
translate charts into data tables before classifying claims and generating
explanations in subsequent steps. With our baselines, we reach $73.9$ (fine-
tuned), $64.2$ (few-shot), and $65.5$ (zero-shot) accuracy in claim
classification.

We also annotate a subset of ChartCheck with $(i)$ reasoning types humans
apply for chart understanding (Amar et al., 2005) and $(ii)$ visual attributes
common in charts (e.g. legends, log/linear scale of axes, labels for data
points, etc.). For example, correctly classifying claim C$1$ in Figure 1
requires _retrieving the value_ for class “Hydro” and _comparing_ it to all
other classes to determine if $36\%$ is the highest one. Moreover, we evaluate
models on reasoning types and chart attributes found in ChartCheck. Our
results suggest that ChartCheck poses a challenging problem and will stimulate
progress on fact-checking against charts.

Our contributions are as follows:

  * •

We introduce ChartCheck, the first chart fact-checking dataset with $1.7k$
real-world charts and $10.5k$ claims and explanations.

  * •

We evaluate chart-to-table and VL baselines in a finetuning and few-/zero-shot
setting.

  * •

We study chart reasoning types and visual attributes that pose a challenge for
SOTA models. Our results suggest the need for new modeling strategies to
address them.

##  2 ChartCheck: Dataset Creation

Figure 2 provides an overview of the entire dataset pipeline. Starting with
chart extraction from the web (step $1$), we applied three crowdsourcing tasks
(step $2-4$), before post-processing and validating the collected data (step
$5$).

###  2.1 Data Collection (Step $1$)

To collect a diverse set of chart images, we used Wikimedia Commons, a project
of the _Wikimedia
Foundation_.111https://commons.wikimedia.org/wiki/Commons:Welcome Wikimedia is
a media repository of images, videos, and audios which are used across all
Wikimedia projects and not subject to copyright restrictions. In Wikimedia
Commons, images are labelled with categories which are part of a multi-
hierarchy structure of categories. Using these labels, we extracted
horizontal/vertical bar charts, line/area charts, pie/donuts charts, and
scatterplots, if the chart description was in English. This step resulted in
$2,498$ total charts.222See Appendix A.1 for further details.

###  2.2 Crowdsourcing (Step $2-4$)

For the subsequent pipeline steps, we used the crowdsourcing platform
_Mechanical Turk_ between Dec $2022$ and May $2023$.333Find annotation UI and
details on worker training and recruitment in Appendix A. Next, we describe
the design of each crowdsourcing task in more detail.

#### 1\. Task: Chart filtering

The aim of the first task was to filter out noisy chart images for the
subsequent tasks. We asked annotators to label if charts were $(i)$ non-
English, $(ii)$ not readable, $(iii)$ not understandable (e.g. because crucial
information was missing), or $(iv)$ showed none of the previous issues. We
filtered the collected data using majority voting, which resulted in $1,684$
charts.

#### 2\. Task: Claim and explanation generation.

For claim and explanation were written by a different group of annotators. We
asked annotators to write one claim which was supported by the chart and its
caption and one refuting claim. The workers also provided a text explaining
why the claims is supported (or refuted, e.g. C$2$ and E$2$ in Figure 1). To
improve the submission quality, we implemented automated checks on the claim
and explanation text. In total, we collected $9,300$ claims and explanations.

Figure 2:  Overview of the ChartCheck dataset pipeline.

#### 3\. Task: Validation of claims and explanation.

Next, we asked another group of workers to verify the claims and explanations
collected in the previous step. If the assigned claim label did not match its
content, workers were asked to correct the claim. Similarly, workers corrected
explanations if they did not explain correctly why a claim was supported (or
refuted) by the chart and its caption. Similar to the first annotation task,
we created sets of seven tasks out of which two where gold-labelled.

#### Training and recruitment of workers.

We allocated the first and last crowdsourcing tasks (i.e. chart filtering and
validation) to three crowdworkers each. Workers with a record of min. $1000$
previously-approved tasks and an approval rate of min. $95\%$ were eligible.
All workers first passed a qualification test by answering multiple-choice
questions given charts. Moreover, we included multiple examples of tasks
labelled by the authors in the instructions and gave rationales for the
correct labels to help workers understand the task. Next, we describe the
design of each crowdsourcing task in more detail.

###  2.3 Quality assurance and validation (Step $5$)

Set | #Charts | #Claims | Support | Refute
---|---|---|---|---
Train | 1,532 | 7,607 | 3,871 | 3,736
Valid | 953 | 672 | 494 | 459
Test $t1$ | 669 | 939 | 487 | 452
Test $t2$ | 151 | 981 | 503 | 478
Total | 1,684 | 10,480 | 5,355 | 5,125

Table 1:  Overview of ChartCheck statistics

To maintain annotation quality and detect malicious behaviour, we included
gold annotated samples in the first and final crowdsourcing task. For the
second annotation task (i.e. claims and explanation generation), we manually
evaluated one claim and explanation per submitted task set, before accepting
the work. We banned workers with malicious behaviour. Moreover, we
automatically checked each claim and explanation for their token length and
further criteria, e.g. ambiguous wording (e.g. mostly, perhaps, and some).

Task | IAA score (R-$\kappa$)
---|---
Chart filtering | 66.2
Claim verification | 61.5
Explanation verification | 51.4

Table 2:  Randolph’s Kappa (R-$\kappa$) as inter-annotator agreement (IAA)
scores.

Inter-annotator agreement (IAA) scores for the claim filtering, claim
verification and explanation verification tasks are are $66.2$, $61.5$, and
$51.4$ respectively (Table 2). We calculated IAA using Randolph’s kappa ($R$).
In case of data imbalance, Fleiss-kappa ($F$) is prone to the high agreement
but low kappa phenomenon (Feinstein and Cicchetti, 1990) and $R$ is preferred.
Agreement over $0.61$ indicates substantial agreement while $51.4$ signifies
moderate agreement according to Landis and Koch (1977).

##  3 Analysis of ChartCheck

###  3.1 Dataset Statistics

Table 1 provides an overview of the ChartCheck dataset. In total, ChartCheck
contains $10,480$ claims, almost equally split in support and refute classes,
and $1,684$ unique charts. In average, the dataset contains six (three
supporting and three refuting) claims per chart image. ChartCheck includes two
testsets: testset $t1$ and $t2$. While $t1$ is a testset with a similar
distribution as the training and validation sets, for $t2$ we only included
chart images which are not part of the training set. With $t2$ we evaluate
models’ performance on charts, which they did not see during training. The
column _#Charts_ states the number of unique charts per dataset sample, which
overlap across dataset splits except testset $t2$, that only contains charts
not occurring in any other split. In average, claims and explanations have a
length of $12.5$ and $23.4$ tokens, respectively.

###  3.2 Chart Attributes

Figure 3:  Chart types in annotated subset. Figure 4:  Reasoning types in
annotated subset.

We manually annotated $100$ charts (i.e. $50$ per testset) with the following
attributes: number of charts per image; chart type; complexity of charts;
presence of datapoint labels, legends, axis titles, and main title; axes
scales; background grid characteristics; font size of title, labels, and other
text occurring in the chart image; visual characteristics (e.g. colors and
positions of legends). Figure 3 gives an overview chart types found in the
annotated subset.

Figure 5:  Baselines Model for ChartCheck.

Out of $100$ annotated chart images, $7\%$ have multiple charts depicted in
one image. The majority of charts (around $34\%$) are pie charts, followed by
bar charts ($29\%$), line charts ($26\%$), $3$D pie charts ($5\%$), and area
charts ($2\%$). Around half of them ($47.5\%$) have labels given for their
data points. Around $38.4\%$ of charts have a legend describing the data
categories. Most of them differentiate between different categories through
colors ($32.3\%$) and only a small portion ($5.1\%$) uses different patterns.
If the chart type has an orientation (e.g. bar charts and line charts), it is
more often horizontal ($63.2\%$) than vertical ($36.8\%$).

###  3.3 Reasoning with Charts

We labelled $230$ dataset samples from both testsets for chart reasoning types
based on a taxonomy of reasoning types humans use while interacting with chart
data (Amar et al., 2005). In total, nine different reasoning types are present
in the labelled subset: find extremum, comparison, world knowledge &
commonsense (KCS), compute derived value, retrieve value, chart features,
multichart reasoning, determine range, and multiaxis. Figure 4 gives an
overview on the reasoning types per testset. Around $100$ claims require
between two to four reasoning types for correct classification of claims.

##  4 Experiments and Results

###  4.1 Baselines

We evaluate several state-of-the-art chart reasoning models on ChartCheck.
Baseline architectures are visualized in Figure 5 and can be grouped in two
categories: $(1)$ VL models using the chart image as input; $(2)$ baselines
translating charts to tables before classification and explanation generation
in subsequent steps.

#### Experimental setup.

We evaluate multi-task baselines which model the explanation generation task
jointly with veracity prediction, as well as baselines with seperate models
for each task. In the multi-task setting, we integrate the explanations in
models’ input prompt and instruct the model to “classify and explain”. In the
single-task setting, we first predict the claim veracity and in a subsequent
step another model generates the explanation using the claim, evidence data,
and either the predicted label or gold label as input. Inspired by the success
of chain of thought prompting (Wei et al., 2022) on complex reasoning tasks,
we include explanations which outline the reasoning steps for claim
classification in the fine-tuning/few-shot examples.444Prompts are given in
the appendix.

Moreover, we evaluate models in three different training settings: fine-tuned,
few-shot, and zero-shot. The fine-tuned models are trained on the entire
training set. For few-shot training, we provide selected training samples per
veracity class (i.e. “support” and “refute”) as input to the models.

We evaluate the veracity classification task with accuracy and macro F$1$. For
explanation generation, we include common natural language generation (NLG)
metrics, i.e. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), BERTScore
(Zhang et al., 2020), METEOR (Banerjee and Lavie, 2005), and BLEURT (Sellam et
al., 2020).

####  $(1)$ Vision-language baseline.

For our vision-language baseline, we use _MatCha_ (Liu et al., 2022b). MatCha
pretrains the image-to-text Pix2Struct model (Lee et al., 2022b) on chart
derendering (chart-to-tables and chart-to-code) and mathematical reasoning. It
achieves state-of-the-art performance on multiple chart datasets, including
ChartQA (Masry et al., 2022), PlotQA v$2$ (Methani et al., 2020), and the Pew
charts subset of Chart-to-text (Kantharaj et al., 2022). Similar to Liu et al.
(2022b), we render a language prompt directly on top of the chart image.

We evaluate three MatCha models in a zero shot setting on our testset: MatCha
fine-tuned on $(i)$ ChartQA (_MatCha-chartqa-zero_), $(ii)$ PlotQA v$1$
(_MatCha-plotqa1-zero_), and $(iii)$ PlotQA v$2$ (_MatCha-plotqa2-zero_). In a
fine-tuned setting, we have MatCha fine-tuning only on the classification task
(_MatCha-finetune-classification_) and in a multi-task setting (_MatCha-
finetune-multi_). Moreover, we experiment with a $50/50$ training data split
during fine-tuning (_MatCha-finetune-50/50-multi_). For the first half of the
dataset, we only fine-tune with classification prompts and for the second half
we include explanations as well to improve overall classification performance.

####  $(2)$ Chart-to-table baseline.

The chart-to-table baselines apply a two-step approach. Inspired by Liu et al.
(2022a), we decompose the chart-based fact-checking task into $(i)$ chart-to-
table conversion and $(ii)$ table-based classification and explanation
generation. First, we translate all images with a zero-shot DePlot base model
into tables.

Using the translated tables and claims, we evaluate the claim classification
task with two models: $(i)$ DeBERTa (He et al., 2021) (_DePlot-DeBERTa-
classification_) and $(ii)$ TAPAS (Herzig et al., 2020) (_DePlot-TAPAS-
classification_). For explanation generation, we fine-tune FlanT$5$ (Shen et
al., 2023) base by inputting the claim, translated table and its caption, and
$(i)$ gold labels (_DePlot-FlanT5-explanation-gold_) or $(ii)$ DeBERTa
classification predictions (_DePlot-FlanT5-explanation-pred-labels_).

Model | Task | Setting | Test$1$ Acc | Test$1$ $F_{1}$ | Test$2$ Acc | Test$2$ $F_{1}$ | Avg Test Acc
---|---|---|---|---|---|---|---
MatCha-chartqa-zero | C | zero | 29.1 | 46.4 | 29.3 | 48.3 | 29.2
MatCha-plotqa1-zero | C | zero | 51.3 | 50.7 | 47.4 | 48.3 | 49.4
MatCha-plotqa2-zero | C | zero | 48.2 | 32.9 | 48.7 | 32.9 | 48.4
MatCha-finetune-class | C | fine-tuning | 64.0 | 63.7 | 61.6 | 60.9 | 62.8
MatCha-finetune-multi | M | fine-tuning | 59.4 | 59.1 | 61.1 | 60.9 | 60.2
MatCha-finetune-50/50-multi | M | fine-tuning | 61.9 | 61.2 | 63.0 | 62.1 | 62.4
MatCha-chartqa-finetune-50/50-multi | M | fine-tuning | 59.1 | 58.8 | 61.4 | 60.7 | 60.2
MatCha-plotqa1-finetune-50/50-multi | M | fine-tuning | 60.6 | 60.5 | 61.6 | 61.4 | 61.1
MatCha-plotqa2-finetune-50/50-multi | M | fine-tuning | 62.8 | 62.8 | 61.4 | 61.4 | 62.1
DePlot-DeBERTa-class | C | fine-tuning | 74.9 | 74.9 | 72.9 | 72.9 | 73.9
DePlot-TAPAS-class | C | fine-tuning | 62.7 | 62.2 | 63.5 | 62.9 | 63.1
DePlot-FlanT5-finetune-class | C | fine-tuning | 69.6 | 69.6 | 67.2 | 67.2 | 68.4
DePlot-FlanT5-zero-multi | M | zero | 65.9 | 64.4 | 65.2 | 64.0 | 65.6
DePlot-FlanT5-few-multi | M | few | 63.2 | 63.0 | 65.2 | 65.1 | 64.2
DePlot-FlanT5-finetune-multi | M | fine-tuning | 66.9 | 66.8 | 67.3 | 67.2 | 67.1

Table 3:  Classification results for model trained on veracity classification-
only (C) and in a multi-task setting with explanation generation (M).

We also test FlanT$5$ in a multi-task setting (i.e. claim classification and
explanation generation). We evaluate a finetuned, (_DePlot-FlanT5-finetune-
multi_), few-shot, (_DePlot-FlanT5-few-multi_), and zero-shot (_DePlot-
FlanT5-zero-multi_) version of the multitask FlanT$5$ model.

Model | Task | Training setting | BLEU | Rouge | MET | Bertscore | BLEURT
---|---|---|---|---|---|---|---
MatCha-finetune-multi | M | fine-tuning | 17.1 | 37.3 | 38.3 | 67.8 | \- 0.48
MatCha-finetune-50/50-multi | M | fine-tuning | 17.3 | 37.1 | 37.9 | 67.3 | \- 0.45
DePlot-FlanT5-zero-multi | M | zero | 0 | 5.04 | 1.3 | 83.2 | -1.48
DePlot-FlanT5-few-multi | M | few | 5.2 | 28.5 | 21.1 | 90.6 | \- 0.56
DePlot-FlanT5-finetune-multi | M | fine-tuning | 17.3 | 46.3 | 36.0 | 91.5 | \- 0.50
DePlot-FlanT5-explanation-gold | E | fine-tuning | 30.7 | 38.7 | 33.2 | 89.6 | \- 0.57
DePlot-FlanT5-explanation-pre-labels | E | fine-tuning | 12.7 | 37.9 | 32.4 | 89.5 | \- 0.6

Table 4:  Explanation generation results for model trained on explanation-only
(expl) and in a multi-task setting with veracity classification (multi).

###  4.2 Results and Discussion

Table 3 and 4 provide an overview of all baselines on claim classification and
explanation generation.

#### What is the best performing baseline

for ChartCheck?

#### Claim Classification.

_DePlot-DeBERTa-class_ achieves with $74.9$ accuracy on $t1$ and $72.9$ on
$t2$, the highest scores on both testsets. The baseline translates chart
images to tables, concatenates the table text into a text sequence, and uses
the claim, table, and the chart caption as input to a DeBERTa large model
fine-tuned on ChartCheck. In general, chart-to-table baselines outperform
their VL counterparts. With $62.8$, the best VL baseline scores around ten
points less. There may be various reasons for difference in performance
though. While DePlot translates charts to tables and DeBERTa performs the
reasoning steps to predict the correct claim label. On the other hand, both
tasks (i.e. data extraction and reasoning) are handled by the MatCha model. We
assume the reduced classification results are related to the MatCha struggling
with reasoning types present in ChartCheck (see Figure 4).

#### Explanation Generation.

We evaluate explanations using five different metrics. _DePlot-
FlanT5-finetune-multi_ outperforms other models on two out of five metrics.
This model uses DePlot chart-to-table translation and finetunes FlanT5-base in
a multitask setting on both claim classification and explanation generation.
It performs best for testset $t1$ based on BERTScore ($91.5$) and ROUGE
($46.3$). For all other metrics, it scores among the top-$3$ best-performing
baselines.

#### What further insights do we gain through

different training settings?

#### Pretraining.

Pretraining MatCha on other chart datasets (e.g. chartQA), has no impact on
models’ performance on ChartCheck. The redundancy of pretraining, can be
related to the datasets available for pretraining. Most of the questions in
ChartQA/PlotQA only require retrieving a single value for the answer.
ChartCheck claims’ are complex and involve filtering, comparing, and
commonsense knowledge among other reasoning types. Moreover, ChartCheck charts
are more diverse (e.g. $3$D pie charts and area charts) compared to charts
found in other datasets.

#### Multitask Learning.

While all best-performing classification models are trained on classification-
only, the multitask setting is more beneficial for explanation generation.
Except for _DePlot-FlanT5-explanation-gold_ and BLEU, multitask models
outperform explanation-only models on all metrics. Another interesting
difference is the impact of zero-/few-shot and finetuning on multitask models.
We discuss this in more details below.

#### Zero and Few shot Evaluation.

Comparing _DePlot-FlanT5-few-multi_ to its zero-shot and finetuned equivalents
(i.e. _DePlot-FlanT5-zero-multi_ and _DePlot-FlanT5-finetune-multi_), we find
contrasting implications for claim classification and explanation generation.
For claim classification, the accuracy for testset $t1$ and $t2$ either do not
change or decreases (from $65.9$ to $63.2$). However, prompting with few
examples improves model’s explanation generation performance across all
reported scores compared to its zero-shot counterpart. Finetuning with more
examples has a larger impact on the generated explanations. Across all
evaluation metrics, the few-shot model yields better explanations than the
zero-shot model. Moreover, finetuning further improves the model’s explanation
quality and gives best results according to Bertscore and ROUGE.

###  4.3 Analysis with Performance Breakdown

#### Which chart types pose challenge for the models?

Table 5 summarizes the performance of the best-performing DePlot and MatCha
baselines. Comparing model performance across chart types, we find both
baselines obtaining relatively high accuracy on pie and bar charts while model
performance on line, 3$D$ pie, donut, and area charts is lower. The difference
between the DePlot- and MatCha-baseline is low for line charts ($62.2$ versus
$59.0$) and donut charts (both reaching $50.0$) but higher for bar and pie
charts. In general, the DePlot baseline performs best on bar charts while
MatCha excels at pie charts compared to other chart types. _DePlot-DeBERTa-
class_ outperforms _MatCha-finetune-class_ on single-chart images reaching
$73.0$ accuracy. Whereas on multi-chart images the DePlot-baseline achieves
only $50.0$ accuracy, less than MatCha ($58.3$). In general, reasoning over
multiple charts present in one image poses a challenge for both.

Interestingly, both baselines struggle with charts that use legends. The
DePlot baseline achieves $75.2$ on charts without legends and $65.3$ if a
legend is given. Similarly, performance drops for the MatCha baseline from
$65.2$ (no legends) to $56.8$ (legends).

#### Do models struggle with specific types of chart reasoning?

While both baselines variants struggle with certain reasoning types, in
general the best-performing DePlot baseline outperform MatCha one across all
reasoning types (see Figure 6). The difference is most significant for the
reasoning types _determine range_ , _compare_ , _chart features_ , and
_multichart_. DePlot correctly predicts all claims requiring to determine a
range of numbers correctly compared to MatCha reaching around $20\%$. For the
other three reasoning types, DePlot correctly predicts $80\%$ or more of all
claims while MatCha accuracy is below $60\%$. Both models struggle with claims
requiring commonsense reasoning (_KCS_).

Figure 6:  Number of correct predictions by reasoning types in labelled subset
with DePlot-DeBERTa-class and MatCha-finetune-multi.

#### Why translating charts to tables with DePlot is challenging?

ChartCheck charts are extracted from the web and miss gold-tables required for
automated evaluation. We manually evaluated a subset of DePlot-generated table
and found the following error cases.555Examples are given in the appendix.
$(1)$ The model hallucinates labels for axes ticks and data points if there
are not present or difficult to read and extract. $(2)$ Errors in table
translation are more prevalent for pie charts than other chart types. $(3)$
Similar colors are confused when plotted against a non-white backgrounds.
$(4)$ Extracting the correct table for grouped bar charts poses a challenge.
$(5)$ Even if written with a large font size and printed as black text on
white background, DePlot often extracts wrong or no titles from the images.
$(6)$ Incorrect text is extracted from charts which have a non-white
background (e.g. grey). $(7)$ In addition to the issues mentioned before,
generated tables sometimes repeat one row multiple times or miss some data
points from the chart (i.e. table rows).

Chart Type | #Claims | DePlot | MatCha
---|---|---|---
Pie | 146 | 76.7 | 66.4
Line | 122 | 62.2 | 59.0
Bar | 104 | 81.7 | 64.4
$3$D Pie | 39 | 66.7 | 39.0
Area | 5 | 60.0 | 40.0
Donut | 2 | 50.0 | 50.0

Table 5:  Performance of DePlot-DeBERTa-class and MatCha-finetune-class across
different chart types.

##  5 Related Works

###  5.1 Datasets for evidence-based fact checking

The goal of evidence-based fact checking is to verify a claim against evidence
data. Typically, evidences are conveyed in different forms such as text
(Thorne et al., 2018; Augenstein et al., 2019; Kotonya and Toni, 2020), tables
(Aly et al., 2021; Akhtar et al., 2022; Chen et al., 2020; Gupta et al.,
2020), and audio/visual materials (Akhtar et al., 2023b). While recent work
introduced a first fact-checking against charts (Akhtar et al., 2023a), the
dataset has certain limitations. First, it is limited to bar charts, ignoring
the diversity of chart types in real-world sources. Second, their charts have
been synthetically created and miss many visual characteristics we find in
real-world charts. Moreover, the claims are extracted from the TabFact (Chen
et al., 2020) dataset and therefore do not require considering visual chart
attributes.

###  5.2 Chart datasets for other tasks

Chart question answering (ChartQA) and chart captioning are two other lines of
research closely related to our task. The former aims to predict the answer by
taking a chart and a natural language question as input. While the task of
ChartQA has received growing attentions in recent years, existing datasets
have several major limitations in terms of size Kim et al. (2021),
automatically generated questions using pre-defined templates Kafle et al.
(2018) or automatically created charts by programming tools Singh and Shekhar
(2020).

Chart captioning refers to generate a caption in natural language text based
on an input chart. Accurate and natural-sounding captions can improve the
accessibility and usefulness of chats. Although, a few prior works focused on
scientific figures Hsu et al. (2021); Chen et al. (2019); Kantharaj et al.
(2022), most of them have tried to generate captions using synthetic images
and text Chen et al. (2019); Kahou et al. (2018). However, synthetic captions
tend to be generic and describe features without conveying higher-level
insights. We address these challenges by proposing a large scale dataset with
real world charts and human created claims.

###  5.3 Modeling approaches for charts

Various approaches have been developed in recent times to address chart
modeling. One category of models encode both chart image and text, and apply a
fusion method (e.g. attention) to combine the features into a joint
representation (Kafle et al., 2018; Masry et al., 2022; Kafle et al., 2020).
Another popular approach is to first extract the data table of the chart,
before applying table-based methods Kim et al. (2020); Liu et al. (2022a);
Methani et al. (2020).

##  6 Insights and Challenges

#### Main Insights.

Experimenting with ChartCheck, we observe that the multi-step, DePlot-based
baselines outperformed the MatCha baselines. Multi-step baselines involve
translating charts to tables before classification and explanation generation.
However, the table translation step in the multi-step baseline is performed by
the DePlot model which shares the same backbone model as MatCha. This and
further analysis indicate that the reasoning types required for ChartCheck
pose a challenge for MatCha.

We observed that employing multitask learning for claims classification and
explanation generation resulted in improved quality of the generated
explanations. However, we did not observe a similar improvement on claim
classification.

The probed models struggle with less common chart types like donut and area
charts, while performing well on simple pie and bar charts. Extracting
information from chart legends is a challenge for both baselines.
Additionally, claims involving world knowledge and commonsense reasoning pose
difficulties as well.

#### Open Challenges.

Based on our experiments and results, we observed the following challenges as
promising directions for future research:

First, there is a need for models that can both extract (visual) information
from charts and successfully solve chart-related tasks that involve various
reasoning skills. Although our DePlot-based baselines performed well,
important visual information is lost during image-to-table translation.

Second, there is a need for future datasets that expand on this work across
various dimensions. One direction to explore is the creation of multilingual
chart datasets. Another direction can consider fact-checking against data
visualization types that are not currently included in ChartCheck, such as
maps and infographics.

Studying charts within the broader context of documents is another interesting
direction to pursue. In real-world settings, charts are often embedded within
documents. Evaluating claims about charts may require combining information
from various sources, such as text, tables, or other images found alongside
the chart.

##  7 Conclusion

This paper introduces ChartCheck, a first dataset for evidence-based fact-
checking with $1.7k$ real-world chart images and $10.5k$ human written claims
and explanations. We evaluate state-of-the-art chart reasoning models, DePlot
and MatCha, in a few-zero-shot and finetuned setting on the proposed dataset
reaching an overall accuracy of $73.9$ in the finetuned setting and
$64.2/65.6$ in the few-/zero-shot case. Moreover, we evaluate the baselines’
performance across reasoning types humans apply to understand chart data and
common chart attributes (chart types, legends, labels, colors, etc.),
highlighting charts and reasoning types the models struggle with. These can be
interesting directions to pursue for future research.

## Limitations

The presented work exhibits the following limitations. First, the it is
limited to English-only, whereas fact-checking misinformation is a global need
that requires solutions in multiple languages. Further research is needed to
extend chart-checking beyond English. Second, although ChartCheck includes a
wide range of charts, certain chart types such as heat maps, scatter plots,
and histograms are missing. Addressing these chart types should be a focus of
future research. Finally, in our attempt to include diverse charts from real-
world sources, we were unable to obtain gold data tables for the charts. In
the future, it would be beneficial to construct a dataset that includes
diverse charts along with their underlying table data. This would enable the
study of a broader range of models and their performance on such data. It
would be also interesting to make the data three way labeled including
"uncertain" label in addition to "true" and "false" claim.

## Ethics Statement

This paper introduces a dataset for automated fact-checking against chart
images. The dataset is intended for research purposes, not as the sole means
of evaluating real-world applications. The labels (i.e. supports and refutes)
describe a claim’s veracity given the evidence table. We do not make any
statement on ChartCheck claims’ truthfulness in a real-world context.

Prior to the crowdsourcing, we obtained ethical clearance from the relevant
authority in our institution. We informed the annotators about all data being
collected and its purpose. Participants had the opportunity to withdraw at any
time and to provide feedback at the end of each task. All annotators were from
English speaking countries. The payment was above the minimum wage and decided
based on the time workers spent on the pilot tasks. In order to support future
research, we plan to release all the scripts and resources used for dataset
creation and model evaluation. This will facilitate and encourage further
research in this field.

## References

  * Akhtar et al. (2022) Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022.  PubHealthTab: A public health table-based dataset for evidence-based fact checking.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1–16, Seattle, United States. Association for Computational Linguistics.
  * Akhtar et al. (2023a) Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2023a.  Reading and reasoning over chart images for evidence-based automated fact-checking.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 399–414, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Akhtar et al. (2023b) Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023b.  Multimodal automated fact-checking: A survey.  _CoRR_ , abs/2305.13507.
  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  FEVEROUS: fact extraction and verification over unstructured and structured information.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Amar et al. (2005) Robert A. Amar, James Eagan, and John T. Stasko. 2005.  Low-level components of analytic activity in information visualization.  In _IEEE Symposium on Information Visualization (InfoVis 2005), 23-25 October 2005, Minneapolis, MN, USA_ , pages 111–117. IEEE Computer Society.
  * Augenstein et al. (2019) Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019.  MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4685–4697, Hong Kong, China. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Carberry et al. (2006) Sandra Carberry, Stephanie Elzer, and Seniz Demir. 2006.  Information graphics: An untapped resource for digital libraries.  In _Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’06, page 581–588, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. (2019) Charles Chen, Ruiyi Zhang, Sungchul Kim, Scott Cohen, Tong Yu, Ryan A. Rossi, and Razvan C. Bunescu. 2019.  Neural caption generation over figures.  In _Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, UbiComp/ISWC 2019 Adjunct, London, UK, September 9-13, 2019_ , pages 482–485. ACM.
  * Chen et al. (2020) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.  Tabfact: A large-scale dataset for table-based fact verification.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Feinstein and Cicchetti (1990) Alvan R Feinstein and Domenic V Cicchetti. 1990.  High agreement but low kappa: I. the problems of two paradoxes.  _Journal of clinical epidemiology_ , 43(6):543–549.
  * Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020.  INFOTABS: Inference on tables as semi-structured data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2309–2324, Online. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: decoding-enhanced bert with disentangled attention.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020.  Tapas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4320–4333. Association for Computational Linguistics.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Hsu et al. (2021) Ting-Yao Hsu, C. Lee Giles, and Ting-Hao Kenneth Huang. 2021.  Scicap: Generating captions for scientific figures.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 3258–3264. Association for Computational Linguistics.
  * Kafle et al. (2018) Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018.  DVQA: understanding data visualizations via question answering.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 5648–5656. Computer Vision Foundation / IEEE Computer Society.
  * Kafle et al. (2020) Kushal Kafle, Robik Shrestha, Brian L. Price, Scott Cohen, and Christopher Kanan. 2020.  Answering questions about data visualizations using efficient bimodal fusion.  In _IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020_ , pages 1487–1496. IEEE.
  * Kahou et al. (2018) Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018.  Figureqa: An annotated figure dataset for visual reasoning.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings_. OpenReview.net.
  * Kantharaj et al. (2022) Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022.  Chart-to-text: A large-scale benchmark for chart summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4005–4023, Dublin, Ireland. Association for Computational Linguistics.
  * Kim et al. (2020) Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. 2020.  Answering questions about charts and generating visual explanations.  In _CHI ’20: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, April 25-30, 2020_ , pages 1–13. ACM.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR.
  * Kotonya and Toni (2020) Neema Kotonya and Francesca Toni. 2020.  Explainable automated fact-checking for public health claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7740–7754, Online. Association for Computational Linguistics.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _Biometrics_ , pages 159–174.
  * Lee et al. (2021) Crystal Lee, Tanya Yang, Gabrielle D. Inchoco, Graham M. Jones, and Arvind Satyanarayan. 2021.  Viral visualizations: How coronavirus skeptics use orthodox data practices to promote unorthodox science online.  In _CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021_ , pages 607:1–607:18. ACM.
  * Lee et al. (2022a) Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022a.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  _CoRR_ , abs/2210.03347.
  * Lee et al. (2022b) Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022b.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  _CoRR_ , abs/2210.03347.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2022a) Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2022a.  Deplot: One-shot visual language reasoning by plot-to-table translation.  _CoRR_ , abs/2212.10505.
  * Liu et al. (2022b) Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel H. Collier, and Julian Martin Eisenschlos. 2022b.  Matcha: Enhancing visual language pretraining with math reasoning and chart derendering.  _CoRR_ , abs/2212.09662.
  * Lo et al. (2022) Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, Enrico Bertini, and Huamin Qu. 2022.  Misinformed by visualization: What do we learn from misinformative visualizations? _Comput. Graph. Forum_ , 41(3):515–525.
  * Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. 2022\.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2263–2279. Association for Computational Linguistics.
  * Methani et al. (2020) Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020.  Plotqa: Reasoning over scientific plots.  In _IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020_ , pages 1516–1525. IEEE.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shen et al. (2023) Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. 2023.  Flan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.  _CoRR_ , abs/2305.14705.
  * Singh and Shekhar (2020) Hrituraj Singh and Sumit Shekhar. 2020.  STL-CQA: Structure-based transformers with localization and encoding for chart question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3275–3284, Online. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and verification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 809–819. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _NeurIPS_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

##  Appendix A Dataset Collection

###  A.1 Wikimedia Commons Repository

We collected charts using Wikimedia Commons, a project of the _Wikimedia
Foundation_ (similar to Wikipedia and
WikiData).666https://commons.wikimedia.org/wiki/Commons:Welcome Wikimedia is a
media repository of images, videos, and audios which are used across all
Wikimedia projects and not subject to copyright restrictions. As of May 2023,
the repository contained over $93$ million files. In Wikimedia Commons, images
are labelled with categories which are part of a multi-hierarchy structure of
categories. We used Wikimedia’s API and the categories
_“Horizontal_bar_charts”_ , _“Vertical_bar_charts”_ , _“Line_charts”_ ,
_“Pie_charts_in_English”_ (e.g. chart in Figure 1), and _“Scatterplots”_ to
extract chart images from the repository. See Figure 11 for a Wikimedia
Commons entry. We only extracted charts, if their description was in English.
We used the spaCy (Honnibal et al., 2020) language detection tool.
Additionally, we extracted the file name, category, description, url, source
(if available), and the Wikipedia pages the image is linked to.

###  A.2 Crowdsourcing

#### Mechanical Turk Details.

For the first and third crowdsourcing task, we created sets of seven
annotation tasks each, out of which two were from a gold-labelled sample set
annotated by the authors.

For the second annotation task (i.e. claim and explanation generation), we
provided annotators a set of seven charts and captions per taskset. For each
chart-caption pair, we asked them to write one claim which was supported by
the chart and its caption and one which was refuted, as well as an
explanation. The claim length was restricted to $[5;30]$ tokens and the
explanation text to $[10;100]$ tokens. To improve the submission quality, we
implemented automated checks on the claim and explanation text.

#### Quality Assurance.

We included gold annotated samples in the first and final crowdsourcing task.
We created a gold standard of $50$ charts for the chart filtering task and
$40$ for the verification task. We used two gold samples per task. Workers who
failed those two samples could not submit their work. For the second
annotation task (i.e. claims and explanation generation), we manually
evaluated one claim and explanation per submitted task set, before accepting
the work. We banned workers with malicious behaviour, e.g. workers who
submitted the same claims as supporting and refuting example. Automatic check
evaluated claims and explanations for their token length, use of ambiguous
wording (e.g. mostly, perhaps, and some), and negation words in case of
refuting claims/explanations.

###  A.3 Dataset Examples

##  Appendix B Modeling Details

#### Technical details.

Before using images as input to MatCha and DePlot we converted a subset of
images which were RGBA images to RGB images by adding a white chart
background. We trained all MatCha models with AdaFactor optimizer, a learning
rate of $1e-5$, a batch size of $4$ and a max patch size of $2048$ for
$15,000$ step. A checkpoint was saved every 300 step and the model with the
best validation loss was selected. We used a max patch size of $2048$. For
DeBERTa-based classification, we evaluated and compared two models, the
DeBERTa base model and a model pretrained on various NLI datasets (e.g. MNLI
and FEVER (Thorne et al., 2018)). We selected the NLI-pretrained DeBERTa model
due to better performance and fine-tuned it with a learning rate of $1e-06$
and a batch size of $4$. We fine-tuned a TAPAS model which was previously
trained on the table fact-checking task TabFact (Chen et al., 2020). The
FlanT$5$ base model was fine-tuned on ChartCheck with a learning rate of
$5e-5$ and batch size of $8$.

#### Instructions for Model Prompting.

For MatCha experiments, our prompts consist of an instruction to the model
with the claim (“[claim]”) and caption (“[caption]”) rendered on top of the
input image. For claim classification tasks, we used the following prompt:
“The chart below has the following caption: [caption]. Given the caption and
the chart below, is the following claim true: [claim]?”. For claim
classification with explanation generation task, we used the following prompt:
“The chart below has the following caption: [caption]. Given the caption and
the chart below, is the following claim true: [claim]? Explain why?”. These
prompts were used for the corresponding task during the _MatCha-
finetune-50/50-multi_ finetuning.

###  B.1 DePlot: Chart-to-table examples

Chart: Table:
| TITLE | Frequency of mobile Wikipedia usage Frequency of mobile Wikipedia usage | in | us At least once a day | 29 | 13 At least once a month | 9 | 15 At least once a week | 36 | 45 Less than once a month | 6 | 10 I never read Wikipedia articles on a mobile device | 5 | 7 |
---|---
Figure 7: Example of DePlot Chart-to-Table conversion Chart: Table:
TITLE | Sources of Electricity Generation New Mexico - 2018 Source of Electricity Generation New Mexico - 2018 | coal | wind | other coal | natural gas | solar 2018 | 0.407 | 0.351 | 0.351 | 0.351 | 0.007 2019 | 0.407 | 0.351 | 0.351 | 0.351 | 0.007 |
---|---
Figure 8: Example of DePlot Chart-to-Table conversion Chart: Table:
TITLE | United States Female Arson Arrests Age | Number arrested 9 | 0.36 10 | 0.82 13 | 2.74 15 | 3.11 16 | 1.83 17 | 2.01 18 | 1.27 19 | 1.44 20 | 1.31 21 | 1.52 22 | 1.62 23 | 1.80 24 | 1.67 25 | 1.69 30 | 1.39 35 | 1.35 40 | 1.33 45 | 0.99 50 | 0.72 55 | 0.52 60 | 0.23 65 | 0.11 |
---|---
Figure 9: Example of DePlot Chart-to-Table conversion Figure 10: Examples of
Charts in the ChartCheck dataset Figure 11:  Chart example from the Wikimedia
Commons page.