#  Simplot: Enhancing Chart Question Answering by Distilling Essentials

Wonjoong Kim1, Sangwu Park211footnotemark: 1, Yeonjun In1, Seokwon Han1,
Chanyoung Park1
1KAIST 2Chung-Ang University
{wjkim, yeonjun.in, dulwich10<EMAIL_ADDRESS><EMAIL_ADDRESS>These
authors contributed equally.Corresponding author.

###### Abstract

Recently, interpreting complex charts with logical reasoning have emerged as
challenges due to the development of vision-language models. A prior state-of-
the-art (SOTA) model, Deplot, has presented an end-to-end method that
leverages the vision-language model to convert charts into table format
utilizing Large Language Models (LLMs) for reasoning. However, unlike natural
images, charts contain a mix of essential and irrelevant information required
for chart reasoning, and we discover that this characteristic can lower the
performance of chart-to-table extraction. In this paper, we introduce Simplot,
a method designed to extract only the elements necessary for chart reasoning.
The proposed method involves two steps: 1) training to mimic a simple plot
that contains only the essential information from a complex chart for table
extraction, followed by 2) performing reasoning based on the table. Our model
enables accurate chart reasoning without the need for additional annotations
or datasets, and its effectiveness is demonstrated through various
experiments. Furthermore, we propose a novel prompt addressing the shortcoming
of recent SOTA model, ignoring visual attributes such as color. Our source
code is available at https://github.com/sangwu99/Simplot.

Simplot: Enhancing Chart Question Answering by Distilling Essentials

Wonjoong Kim1††thanks: These authors contributed equally., Sangwu
Park211footnotemark: 1, Yeonjun In1, Seokwon Han1, Chanyoung Park1††thanks:
Corresponding author. 1KAIST 2Chung-Ang University {wjkim, yeonjun.in,
dulwich10<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

The rapid advancements in vision-language models have accelerated a wave of
research into models capable of handling data that integrates both images and
text (Zhang et al., 2021; Zhou et al., 2020; Kim et al., 2023, 2024), thus
undertaking a variety of tasks. Among these tasks, the interest in models
capable of advanced reasoning, including question answering, has significantly
increased. This increasing field has seen considerable success in addressing
visual question answering (VQA) tasks targeted at natural images, marking a
trend towards models that can engage in complex reasoning based on images
(Antol et al., 2015; Shao et al., 2023; Gardères et al., 2020). Despite these
advancements, the domain of mathematical multimodal reasoning such as
interpreting charts remains relatively unexplored. Mathematical reasoning in
question answering poses unique challenges, as models proficient in natural
images struggle with specific types, such as charts. Charts, with their unique
formats and the need for logical interpretation, necessitate a different
approach to learning compared to conventional VQA models targeting natural
images.

Prior chart reasoning methodologies mainly rely on heuristic rule-based
systems, and thus they are not only limited to pre-defined chart formats but
also struggle with novel chart types without additional rule formulation (Luo
et al., 2021). Moreover, the performance of models utilizing OCR or key-point
detection modules are highly dependent on the performance of these modules,
and also face significant annotation costs and are typically unable to perform
end-to-end reasoning (Methani et al., 2020; Poco and Heer, 2017). In response
to these limitations, recent approaches have adopted vision-language models
trained in an end-to-end manner without heuristic rules (Cheng et al., 2023;
Liu et al., 2022b), which, however, require fine-tuning for each specific
downstream task, limiting their flexibility.

A novel approach, called Deplot (Liu et al., 2022a), that combines vision-
language models with Large Language Models (LLMs) has emerged as an approach
to address these issues. Deplot first transforms charts into tables (i.e.,
chart-to-table extraction), and then employs the extracted tables along with
the LLMs for reasoning. This approach not only aims to solve the inherent
problems of previous methodologies, but also leverages the capability of the
LLMs to enhance performance in chart question answering. Converting charts
into tables before reasoning offers several advantages, including improved
interpretability and the ability to achieve high performance in table
reasoning, thus facilitates more accurate and delicate reasoning compared to a
direct image-based question answering.

Figure 1: An example of chart-to-table extraction (from (a) to (b)) by Deplot
(Liu et al., 2022a) in the ChartQA dataset. Deplot fails to incorporate
textual information within charts (ex."million / billion") and extract
unnecessary information from real-world charts (ex."CC BY") instead of
necessary information (ex. "Northern America"), leading to inaccurate
reasoning.

Despite the effectiveness of Deplot (Liu et al., 2022a), there exist two
limitations that still need to be addressed. First, we discovered that Deplot
struggles to fully utilize the textual context within the image containing the
chart for chart-to-table extraction (See Fig. 1). For example, the counting
unit (e.g., “million” or “billion”) is overlooked when converting a chart
(Fig. 1(a)) into a table (Fig. 1(b)). Moreover, irrelevant information (e.g.,
“CC BY”) is extracted as an entity in the table, while relevant information
(e.g., “Northern America”) is overlooked. However, as real-world charts often
contain information that might not be helpful for chart reasoning (e.g.,
source credit and “CC BY”), it is crucial for the model to be able to
differentiate between relevant and irrelevant information. Owing to these
challenges, Deplot is prone to extracting inaccurate values, leading to
failure to effectively interpret the information of the chart.

The second limitation of Deplot is that it solely relies on tables to solve
chart reasoning tasks, while overlooking the visual information associated
with the tables. For this reason, Deplot fails to answer questions regarding
color or position (e.g., "What is the value of the third bar from the top?",
"What year does the orange line represent?"), indicating a significant
shortfall in the model’s capability to interpret visual data.

To address these two limitations of Deplot, we propose a simple yet effective
method, named Simplot. The main idea is to pre-extract the textual information
within an image containing a chart to provide a guideline for the model
regarding the columns and rows of the table it needs to construct. More
precisely, we divide the training process of Simplot into two phases. In Phase
1, we first convert the original chart, which often contains irrelevant
information regarding the chart reasoning task, into a simple chart containing
only the essential information for chart reasoning. Then, we train a teacher
model to generate a table given the simple chart as input. During Phase 2, the
student encoder is trained to ensure that the representation of an original
chart image closely matches the representation of its corresponding simplified
chart. This capability enables the student encoder to encode any original
chart into the representation space of simplified chart images, thereby
improving the accuracy of table generation. Furthermore, we leverage a Large
Multimodal Model (LMM) to utilize the visual attributes of a chart such as
colors and positions, and present a novel prompt for specifically designed for
chart reasoning.

Our contributions are summarized as follows:

  1. 1.

We provide guidelines for utilizing textual information within images for
reasoning that was previously overlooked.

  2. 2.

Simplot extracts only essential information from complex charts, preventing
irrelevant information from entering the model, resulting in detailed
reasoning .

  3. 3.

We present a prompt specifically designed for chart reasoning. Through this
prompt, the LMM performs more accurate reasoning by mimicking how humans
interpret charts.

  4. 4.

Various experiments and case studies validate that Simplot successfully
addresses the two limitations of Deplot, showcasing significant improvements
in both chart interpretation and reasoning capabilities.

##  2 Proposed Method: Simplot

Figure 2: Overall framework of Simplot. Upper box presents the preprocessing
stage, which involves generating a positive sample containing only essential
information from the original chart, as well as a negative sample, along with
row and column rendering. Lower left box illustrates the training stage
including two phases and the right box describes the inference stage. In Phase
1 of the training stage, a teacher encoder and a table decoder are trained
using a simple chart, and in Phase 2, a student encoder is trained with the
original chart, while being distilled the knowledge of the teacher encoder on
how to generate a table from a simple chart. Lower right box illustrates the
inference stage, where an LMM receives the original chart and the extracted
table along with prompts for reasoning.

In this section, we describe our proposed method, Simplot, in detail. Fig 2
presents the overall framework of Simplot, where Simplot eventually performs
reasoning with an LMM given the table extracted from a chart (Inference
stage). In this work, we focus on chart-to-table extraction (Training stage)
as a high-quality table enables precise reasoning with a powerful LMM. It is
important to note that Simplot can be combined with diverse LMM variants, and
various techniques such as Chain-of-Thought (CoT) (Wei et al., 2022) can
further improve the accuracy of the chart reasoning.

###  2.1 Preprocessing Stage

Before the training stage, we conduct two preprocessing steps, i.e., 1) Simple
Chart Generation, and 2) Row-Column Rendering. Note that as these steps can be
readily done offline, they don’t increase the training time.

1) Simple Chart Generation.  We generate a simple chart by excluding
irrelevant information from the original, keeping only essential elements for
reasoning. Specifically, since each of the original charts in the dataset is
annotated with a table in the CSV format, we use a Python library (i.e.,
Matplotlib) to plot a chart based on the table. The chart generated in this
manner is considered as a simplified version of the original chart, i.e.,
simple chart. This process requires no separate training or additional costs,
and it can be executed offline by running a simple code snippet.

2) Row-Column Rendering.  Inspired by rendering questions over images used for
QA (Lee et al., 2023), we aim to improve the table extraction accuracy by
rendering information about the rows and columns that should be included in
the table onto the image, enabling the model to utilize this information to
extract more accurate tables when converting charts to tables. Note that in
the training stage, since we are given the ground-truth chart-table pairs, we
simply render rows and columns of each table onto its paired image containing
the chart. However, it would be not feasible in the inference stage, since we
would be only given the charts without tables. Instead, in the inference
stage, we utilize a LMM to extract rows and columns from the chart and render
them onto the image. Despite the relative low performance of LMMs such as
GPT-4 (Achiam et al., 2023) in reasoning about charts, we found they can
accurately extract row and column information from charts, since such
information is given as text and structured in a relatively simple manner.
Detailed description of the process is presented in Appendix A.

###  2.2 Training Stage: Chart-to-Table Extraction

Our proposed chart-to-table extraction approach consists of two phases: Phase
1) Training a teacher encoder and a table decoder by performing the chart-to-
table extraction given simple charts containing only the essential information
for reasoning, rather than complex original charts; Phase 2) Training a
student encoder via contrastive learning by extracting the table given the
original chart, while being distilled the knowledge to embed original charts
to the embedding space of simple charts.

####  2.2.1 Phase 1: Learning with Essential Part from Simple Chart

Our model learns the process of extracting tables from previously generated
simple charts. Specifically, we use Deplot Liu et al. (2022a) as our backbone
model, which is our baseline model that consists of an image encoder and a
text decoder, and fine-tune it on the generated simple chart-table pairs. This
facilitates the image encoder to obtain representations containing only
essential information within the chart, while the text decoder converts this
representation into a table format. In this paper, we name the encoder and
decoder as chart encoder $E_{chart}$ and table decoder $D_{table}$,
respectively.

To support our model design that focuses on the essential parts when learning
from charts, in Fig. 3, we compare the chart-to-table extraction performance
of using two types of data for both training and inference, i.e., 1) original
chart (in green), and 2) simple chart containing essential information only
(in red). We observe that using simple charts for chart-to-table extraction
greatly outperforms the case when original charts are used. The chart encoder
trained here serves as the teacher encoder $E_{chart}^{teacher}$, providing
guidelines for the representation that the student encoder
$E_{chart}^{student}$ should learn in the subsequent learning process to be
described in Sec. 2.2.2. Following Deplot, we adopt ViT (Dosovitskiy et al.,
2020) as the chart encoders. Note that the trained table decoder $D_{table}$
and a fully connected layer ($FC$) remain frozen without further training in
the next stage to generate tables.

Figure 3: Accuracy of chart-to-table extraction using the original (in green)
and the simple chart (in red).

####  2.2.2 Phase 2: Encoding Original Chart into Representation Space of
Simple Chart

As shown in Fig. 2, after training the teacher encoder $E_{chart}^{teacher}$
with simple charts as inputs, our goal is to train the student encoder
$E_{chart}^{student}$ in a way that the representation of an original chart
image obtained from $E_{chart}^{student}$ closely matches the representation
of its corresponding simplified chart obtained from $E_{chart}^{teacher}$.
This is to enable $E_{chart}^{student}$ to encode any original chart into the
representation space of simplified chart images, thereby improving the
accuracy of table generation. Then, given the chart representations, we
generate table representations using the frozen table decoder $D_{table}$ that
is trained in Phase 1. More precisely, given an original chart $A$ and the
simplified chart $P$ as the positive sample, we generate a negative sample $N$
by randomly shuffling values in the simple chart $P$. Then, we define a
triplet loss using the representations of the original chart $z_{a}$, its
corresponding positive sample $z_{p}$, and negative sample $z_{n}$ obtained
from the sequentially passing $A,P$ and $N$ into chart encoders and the table
decoder $D_{table}$ as follows:

$\begin{split}\mathcal{L}_{triplet}(A,P,N)=&max\\{d(z_{a},z_{p})\\\
&-d(z_{a},z_{n})+m,0\\},\end{split}$ (1)
$\begin{array}[]{cl}&z_{a}=D_{table}(E_{chart}^{student}(A)),\\\\[4.30554pt]
\text{where}&z_{p}=D_{table}(E_{chart}^{teacher}(P)),\\\\[4.30554pt]
&z_{n}=D_{table}(E_{chart}^{teacher}(N)).\\\ \end{array}$

where $d$ denotes the distance defined as
$d(z_{i},z_{j})=\|z_{i}-z_{j}\|_{2}$, and $m$ denotes the margin which induces
the anchor $z_{a}$ to be closer to the positive sample $z_{p}$ and farther
from the negative sample $z_{n}$. Through this process, the student chart
encoder $E_{chart}^{student}$ is trained to extract only the essential
information from original charts, even if they contain information that is
irrelevant to chart reasoning. By optimizing Eq. 1, $E_{chart}^{student}$
focuses on subtle differences in the representations of positive and negative
samples that would be highly similar in the representation space, which is
expected to enhance the precise value mapping for table generation.

The representation $z_{a}$ obtained after passing the original chart $A$
through the student encoder $E_{chart}^{student}$ and the table decoder
$D_{table}$ is then fed into the frozen fully connected layer ($FC$) trained
in Phase 1 to generate a table.

Furthermore, since our goal is not just to train $E_{chart}^{student}$ to
mimic the representation of simple charts, but to eventually generate tables
effectively, we also introduce a cross-entropy loss for table generation as
follows:

$T=[\hat{y}_{1},\ldots,\hat{y}_{N}]=FC(z_{a}),$ (2)
$\mathcal{L}_{table}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}y_{i,c}\log\left(\frac{\exp(\hat{y}_{i,c})}{\sum_{j=1}^{C}\exp(\hat{y}_{i,j})}\right),$
(3)

where $T$ a linearized textual sequence of the the generated table, $N$ and
$C$ denote the length of the linearized textual sequence of the table $T$, and
the number of classes, respectively, and $y_{i}$ and $\hat{y}_{i}$ denote the
ground-truth token and the predicted token each belonging to one of the $C$
classes, i.e., $y_{i,c}=0$ if $y_{i}$ belongs to class $c$ and 0 otherwise.
Note that, as in Deplot, lines of $T$ are separated by ’<0x0A>’ (line break
symbol), and cells distinguished by ’|’.

The final loss function for chart-to-table extraction is defined with
hyperparameter $\lambda$ to balance two losses as follows:

$\mathcal{L}_{final}=\lambda\mathcal{L}_{triplet}+(1-\lambda)\mathcal{L}_{table}.$
(4)

Note that considering the scale of the two losses, i.e.,
$\mathcal{L}_{triplet}$ and $\mathcal{L}_{table}$, we set $\lambda$ to 0.1 to
balance between them. However, the final performance is expected to improve
when $\lambda$ is carefully selected, which we leave as future work.

In summary, in Phase 2, we align the representation space of
$E_{chart}^{student}$, which is trained with original charts, and
$E_{chart}^{teacher}$, which is trained with simple charts, thereby allowing
$E_{chart}^{student}$ to generate chart representations that mainly contain
essential information for chart reasoning. Based on the representations
obtained from $E_{chart}^{student}$, we use the frozen decoder $D_{table}$ to
generate tables that will be passed on to the LMM in the inference stage.

###  2.3 Inference Stage: Reasoning with Extracted Table

In the inference stage, we use a Large Multimodal Model (LMM) to perform
various reasoning tasks given the tables obtained from the training stage as
input. However, relying solely on tables without the associated visual
information to solve chart reasoning tasks makes it impossible for the model
to answer questions regarding color or position (e.g., "What is the value of
the third bar from the top?", "What year does the orange line represent?"). To
address this issue, we additionally provide the original chart as another
input to the LMM to enable the model to provide answers to a wider range of
questions. By doing so, we expect the model to not only answer to questions
regarding precise chart values by referring to the tables, but also to answer
about visual attributes not contained in the table by referring to the
original chart.

Figure 4: Snippet of proposed prompt.

Human-oriented chart instruction.  Furthermore, we present a novel prompt that
is specifically designed for chart reasoning, named Human-oriented chart
instruction. Although prompt engineering is a widely studied topic for
processing natural images (Kim et al., 2023; Li et al., 2022), it is crucial
to develop specialized prompts specifically tailored for charts, due to the
inherent nature of chart reasoning tasks, i.e., they require more advanced
advanced thinking steps for interpretation. Our proposed prompt provides
instructions that mimic how humans interpret charts, enabling more accurate
reasoning within the LMM. To our best knowledge, this is the first work to
study a prompt specifically targeting charts, and its effectiveness is will be
demonstrated in Section 3.2.2. We provide a snippet of our proposed prompt in
Fig. 4. For detailed description of Human-oriented chart instruction, please
refer to Appendix B.

##  3 Experiments

###  3.1 Experimental Setting

Dataset. In this paper, we evaluate Simplot using a widely used dataset,
ChartQA (Masry et al., 2022) comprising three types of charts (i.e., pie, bar,
and line). ChartQA also includes human-authored and LLM-augmented QA pairs.
Detailed statistics of ChartQA dataset are presented in Table 1. Please note
that we do not use QA pairs for training, but train the model to generate
tables only from the charts in the training set, then use QA pairs from the
test set for measuring reasoning performance. Further description of ChartQA
dataset is presented in Appendix C.1.

Table 1: Statistics of ChartQA dataset.

split type | Pie | Bar | Line | QA pair
---|---|---|---|---
Train set | 541 | 15,581 | 2,195 | -
Validation set | 48 | 837 | 171 | -
Test set | 78 | 1,230 | 211 | 2,500

Baselines. In this paper, we use various models as baselines that can perform
chart reasoning. The baselines we employ are divided into three categories
including Vision-Language pre-trained models (VLP), fully supervised models
for chart reasoning, few-shot model which extracts table and reasoning using
LLM. Detailed descriptions of the models are provided in the Appendix C.2.

Evaluation protocol. For question answering based on charts, we report Relaxed
Accuracy (RA) of 2,500 questions in the test set following previous works (Lee
et al., 2023; Kantharaj et al., 2022b; Masry et al., 2022; Kantharaj et al.,
2022a; Liu et al., 2022b).

Furthermore, we propose a proper metric to evaluate for chart extraction,
employing a slightly modified metric derived from Relative Mapping Similarity
($RMS$), presented by Deplot (Liu et al., 2022a). We discover that $RMS$ does
not accurately measure the chart-to-table performance in certain cases, we
adopt a metric named Relative Distance ($RD$). For the differences between
$RMS$ and $RD$, please refer to the Appendix C.3.

###  3.2 Experimental Results

####  3.2.1 Results on Chart-to-Table Extraction

In Table 2, we observe that Simplot exhibits superior chart-to-table
extraction overall performance compared to existing methods including Deplot
(Liu et al., 2022a) and UniChart (Masry et al., 2023). We argue that the
superiority of Simplot in terms of chart-to-table extraction eventually leads
to more precise reasoning as will be shown in Section 3.2.2.

Furthermore, to verify whether the proposed method demonstrates consistent
performance regardless of chart type, we also evaluate the table extraction
performance separately for each chart type. Except when comparing line
performance with UniChart, Simplot consistently outperforms the baselines in
all cases, with a marginally lower performance (0.42%) observed compared to
UniChart specifically in line charts. However, considering the higher overall
performance of Simplot and the fact that UniChart was pre-trained on over
600,000 data samples for chart-to-table extraction while we use a model pre-
trained on approximately 60,000 data samples (Liu et al., 2022a), it is
evident that our method is sufficiently effective. Moreover, our approach can
be applied to any model pre-trained on large vision-text datasets, and there
is room for performance improvement if we utilize more dataset.

Table 2: Chart-to-table extraction performance ($RD_{F1}$) on the ChartQA
dataset over various chart types.

Models | Chart type | Overall
---|---|---
Pie | Bar | Line
UniChart | 84.86 | 92.58 | 85.16 | 88.03
Deplot | 88.82 | 96.37 | 82.25 | 90.95
Simplot | 91.41 | 96.87 | 84.74 | 92.32

####  3.2.2 Results on Chart Reasoning

The performance of question answering on ChartQA datasets are summarized in
Table 3. We make the following key observations:

(1) In general, we observe that methods designed with a focus on chart
reasoning outperform VLP in QA tasks in terms of overall performance. This
demonstrates that despite the superior performance of vision-language models
in performing a wide range of tasks in the traditional natural image domain,
they face difficulties in interpreting charts due to the significantly
different characteristics of natural images and charts. This highlights the
need for research targeted at chart reasoning by focusing on the
characteristics of charts.

(2) Furthermore, noticeable performance difference is observed among models
aimed at chart reasoning. Methods using tables such as Deplot (Liu et al.,
2022a) or our proposed method, Simplot, generally outperform models that
conduct reasoning based solely on images. This suggests that going through
tables allows for more detailed reasoning, aligning well with our motivation
that emphasizes the importance of effectively extracting tables. Additionally,
we would like to highlight that while other models are fine-tuned with fully
supervision for question answering, Deplot (Liu et al., 2022a) or Simplot are
only trained for table extraction, but they show higher performance even with
few-shot reasoning. While models fine-tuned for chart reasoning need
appropriate data and a new fine-tuning stage to perform other tasks such as
text summarization, Simplot can efficiently handle various tasks without any
additional tuning by utilizing extracted tables and LLM.

Table 3: Chart question answering performance on the ChartQA dataset.

| Models | Data type
---|---|---
| Human | Augmented | Overall
VLP models | TaPas | 28.72 | 53.84 | 41.28
V-TaPas | 29.60 | 61.44 | 45.52
T5 | 25.12 | 56.96 | 41.04
VL-T5 | 26.24 | 56.88 | 41.56
PaLI | 30.40 | 64.90 | 47.65
Mini-GPT | 8.40 | 15.60 | 12.00
LLaVa | 37.68 | 72.96 | 55.32
GPT-4V | 56.48 | 63.04 | 59.76
Supervised | ChartQA | 40.08 | 63.60 | 51.84
ChartT5 | 31.80 | 74.40 | 53.10
Pix2Struct | 30.50 | 71.60 | 56.05
MatCha | 38.20 | 90.20 | 64.20
Unichart | 43.92 | 88.56 | 66.24
ChartLlama | 48.96 | 90.36 | 69.66
Few-shot | Deplot | 62.71 | 78.63 | 70.67
Deplot + img. | 72.39 | 85.01 | 78.70
Simplot | 73.91 | 85.67 | 79.79
Simplot++ | 78.07 | 88.42 | 83.24

(3) Simplot outperforms Deplot (Liu et al., 2022a), which is the SOTA model.
For a more fair comparison, we also report the performance when both the table
extracted by Deplot and the original chart were provided as inputs to an LMM
for reasoning (Deplot+img. in Table 3). We confirm that incorporating images
along with tables can enhance reasoning performance compared with when solely
relying on the tables, again highlighting our motivation, i.e., considering
visual attributes of charts is crucial. The superior performance of Simplot
compared to Deplot+img. is attributed to the effectiveness of our table
extraction method based on contrastive learning. It is important to note that
the performance difference solely originates from the chart extraction
performance since the same prompts were used for both Deplot and Simplot for a
fair comparison, without using our Human-oriented chart instruction.

(4) In Table 3, we observe that Simplot++, which is our model with the, Human-
oriented chart instruction, outperforms vanilla Simplot. This confirms the
importance of using appropriate prompts tailored to the task when employing
LMMs, while also indicating how well our prompt mimics how humans think. It is
notable that providing more diverse in-context examples could lead to even
greater performance gains.

Discussion. The overall performance of Simplot is consistently higher than
other baselines; however, on the augmented set, it sometimes exhibits slightly
lower performance compared to some models. This can be attributed to the
characteristic of the augmented set, which is obtained by fine-tuning the T5
model on the SQuAD QA dataset (Rajpurkar et al., 2016) to generate questions
and answers. Consequently, the questions in the augmented set are mostly
simpler and relatively easier to answer compared to those in the human set.
Fine-tuned models on such questions with full supervision tend to show notably
higher performance on augmented data. For instance, MatCha (Liu et al., 2022b)
shows a performance of 90.2 on the augmented set, but its performance drops
significantly to 38.2 on the human set which is composed of more complex QA
pairs, resulting in an overall performance of 64.2. In contrast, Simplot’s
high performance even on the human set without additional fine-tuning suggests
that proposed method is effective for more complex reasoning tasks.

Figure 5: Case study. Capture contextual information (billion/million) and
extract precise values in the chart.

####  3.2.3 Case Study

We provide specific cases of how Simplot addresses the concerns we have
raised, compared to the our main baseline, Deplot (Liu et al., 2022a).

(1) Firstly, we verify how our method addresses the issue of existing methods
failing to properly reflect textual information within charts. Fig. 5
illustrates that the table extracted by Deplot (Liu et al., 2022a) fails to
consider the counting unit, i.e., "million/billion", present in the chart,
resulting in an inaccurate table. In contrast, Simplot generates a table that
incorporates this information, enabling more accurate reasoning.

(2) Furthermore, we confirm that the process of rendering columns and rows of
a table on the image to provide guidelines for table extraction is beneficial
for extracting accurate an table. In Fig. 6, Deplot (Liu et al., 2022a) omits
a relevant row, i.e., "Montenegro", redundantly extracts several rows, and
even generates entirely irrelevant rows such as "CC BY," which would result in
a poor performance in reasoning tasks. In contrast, Simplot accurately
generates the table by extracting only the relevant rows, ensuring accurate
reasoning.

(3) Lastly, we demonstrate the effectiveness of our proposed prompt, Human-
oriented chart instruction. Fig. 7 illustrates the difference between
interpreting the same chart with/without using the proposed prompt. While the
model fails to produce an accurate answer without the prompt, the model
generates a correct answer in a process similar to how humans interpret charts
when using the prompt. This highlights the effectiveness of our prompt
designed to mimic human reasoning. Please refer to Appendix D for further case
studies of other chart types.

Furthermore, we present typical errors that cause our model to fail in chart-
to-table extraction or question answering, along with possible solutions, in
the appendix E.

Figure 6: Case study. Extract precise rows without unnecessary information in
the chart. Figure 7: Case study. Interpreting charts similar to how humans do
using Human-oriented chart instruction.

##  4 Related Work

Pre-trained vision-language models. Vision-language pre-training seeks to
enhance the performance of downstream vision and language tasks such as visual
question answering (VQA) (Tan and Bansal, 2019; Alayrac et al., 2022), image-
text retrieval (Chen et al., 2020; Jia et al., 2021), and image captioning (Li
et al., 2020, 2022).

Several models learn cross-modal representation from visual features and
language tokens (Chen et al., 2020; Su et al., 2019). Specifically, Lu et al.
(2019) employs two single-modal networks, alongside a cross-modal transformer
layer integrating information from input sentences and images.

Radford et al. (2021) and Jia et al. (2021) facilitated zero-shot adaptation
utilizing contrastive learning for various tasks on multimodal data. However,
their drawback resides in their limitation to generate language, diminishing
their suitability for open-ended tasks such as VQA.

Others propose a different approach to develop an enhanced Large Multimodal
Model (LMM). For example, Liu et al. (2023) employs a visual encoder and a
language decoder from other studies (Radford et al., 2021; Chiang et al.,
2023). It is capable of performing various QA tasks including conversation,
detail description, complex reasoning and focuses on chat capability.

Chart reasoning models. Early methods for processing charts predominantly
utilized rule-based approaches (Balaji et al., 2018; Choi et al., 2019).
However, they were specifically tailored to predefined chart formats, limiting
their applicability only to certain types of charts. Moreover, incorporating
new chart designs require additional rules, leading to the impracticality of
immediate adaptation to new chart formats.

Due to above limitations, researchers utilize modules such as OCR and key-
point detection (Poco and Heer, 2017; Zhou et al., 2023; Xue et al., 2023).
However, these methods can be time-consuming and require dataset annotation
for labeling, and rely on the performance of modules. For example, Luo et al.
(2021) suffers from a drawback as it solely predicts the raw data values
without establishing connections to their respective axes or legends.

In an effort to address these challenges, researchers develop vision-language
models (Meng et al., 2024; Huang et al., 2023) without heuristic rules. Some
approaches comprehend the chart and respond to questions in natural language
(Masry et al., 2023; Han et al., 2023). However, such models require fine-
tuning for each downstream task, constraining their adaptability for diverse
tasks. Meanwhile, studies such as Methani et al. (2020); Lu et al. (2023a)
have created datasets to enable more realistic mathematical reasoning.

To tackle these challenges, recent studies Liu et al. (2022a); Xia et al.
(2023) introduce a two-step approach, chart-to-table extraction and reasoning
with LLM. Recent advancements like Liu et al. (2022a); Xia et al. (2023) have
introduced a two-step approach to tackle challenges in chart analysis.
Firstly, charts are transformed into tables, improving interpretability and
aiding in identifying inaccuracies. Lu et al. (2023b) utilizes Program of
Thought (PoT), achieving significant performance gains. This approach enables
more accurate and interpretable reasoning compared to traditional methods,
indicating that table extraction is crucial.

##  5 Conclusion

In this paper, we propose Simplot, which extracts only the essential
information required for chart reasoning from real-world charts containing a
lot of unnecessary information. Focusing on essential information in charts,
our model extracts precise tables from charts, resulting in accurate chart
reasoning. Furthermore, by leveraging textual information and visual component
such as colors inside the charts, Simplot enables accurate chart reasoning,
which is crucial point but ignored in existing methods. Finally, we propose a
novel prompt, Human-oriented chart instruction, specifically designed for
chart reasoning. It provides instruction to LMM imitating how humans interpret
chart, enabling more precise reasoning. Through extensive experiments and case
studies, we demonstrate that Simplot effectively handle concerns raised by
existing works while improving chart reasoning performance.

## Limitations

Despite the outstanding performance of Simplot in chart-to-table extraction,
there is room for improvement. During the process of rendering columns and
rows, inaccuracies in extracting columns and rows can lead to failures in
effectively converting charts into tables. Particularly, it can be challenging
to extract appropriate columns and rows when the texts in the chart are
complex. We provide a detailed error analysis related to this issue in the
Appendix E. To address these problems, future work could involve refining the
extraction process by understanding the context between extracted columns and
rows before rendering.

## Ethics Statement

Simplot has not been trained with private or sensitive data. This
significantly reduces the risk of generating harmful or misleading content.
The ChartQA dataset, which we utilize to collect chart-table pairs, is
publicly accessible for research purposes. To make sure the transparency and
reproducibility of our experiments, we provide detailed information on
hyperparameter configurations in our paper and publicly share our source code.
This careful approach mitigates the potential for unethical outcomes
associated with data usage. While our models demonstrate state-of-the-art
performance on ChartQA, however, we acknowledge the possibility of their
misuse. There exists a risk that our models could be exploited to mislead the
public about the content and implications of charts. Despite our models’ high
performance, we cannot guarantee their outputs will always be accurate,
emphasizing the need for critical interpretation and verification of results.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433.
  * Balaji et al. (2018) Abhijit Balaji, Thuvaarakkesh Ramanathan, and Venkateshwarlu Sonathi. 2018.  Chart-text: A fully automated chart image descriptor.  _arXiv preprint arXiv:1812.10636_.
  * Biten et al. (2019) Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Minesh Mathew, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. 2019.  Icdar 2019 competition on scene text visual question answering.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1563–1570. IEEE.
  * Bujard et al. (1987) Hermann Bujard, Reiner Gentz, Michael Lanzer, Dietrich Stueber, Michael Mueller, Ibrahim Ibrahimi, Marie-Therese Haeuptle, and Bernhard Dobberstein. 1987.  [26] a t5 promoter-based transcription-translation system for the analysis of proteins in vitro and in vivo.  In _Methods in enzymology_ , volume 155, pages 416–433. Elsevier.
  * Chen et al. (2022a) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022a.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_.
  * Chen et al. (2022b) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022b.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer.
  * Cheng et al. (2023) Zhi-Qi Cheng, Qi Dai, Siyao Li, Jingdong Sun, Teruko Mitamura, and Alexander G Hauptmann. 2023.  Chartreader: A unified framework for chart derendering and comprehension without heuristic rules.  _arXiv preprint arXiv:2304.02173_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  In _International Conference on Machine Learning_ , pages 1931–1942. PMLR.
  * Choi et al. (2019) Jinho Choi, Sanghun Jung, Deok Gun Park, Jaegul Choo, and Niklas Elmqvist. 2019.  Visualizing for the non-visual: Enabling the visually impaired to use visualization.  In _Computer Graphics Forum_ , volume 38, pages 249–260. Wiley Online Library.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Gardères et al. (2020) François Gardères, Maryam Ziaeefard, Baptiste Abeloos, and Freddy Lecue. 2020.  Conceptbert: Concept-aware representation for visual question answering.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 489–498.
  * Han et al. (2023) Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023.  Chartllama: A multimodal llm for chart understanding and generation.  _arXiv preprint arXiv:2311.16483_.
  * Holmgren et al. (2012) Johan Holmgren, Paul Davidsson, Jan A Persson, and Linda Ramstedt. 2012.  Tapas: A multi-agent-based model for simulation of transport chains.  _Simulation Modelling Practice and Theory_ , 23:1–18.
  * Huang et al. (2023) Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. 2023.  Do lvlms understand charts? analyzing and correcting factual errors in chart captioning.  _arXiv preprint arXiv:2312.10160_.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR.
  * Kafle et al. (2018) Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018.  Dvqa: Understanding data visualizations via question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5648–5656.
  * Kantharaj et al. (2022a) Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022a.  Opencqa: Open-ended question answering with charts.  _arXiv preprint arXiv:2210.06628_.
  * Kantharaj et al. (2022b) Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022b.  Chart-to-text: A large-scale benchmark for chart summarization.  _arXiv preprint arXiv:2203.06486_.
  * Kim et al. (2024) Kibum Kim, Kanghoon Yoon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. 2024.  Adaptive self-training framework for fine-grained scene graph generation.  _arXiv preprint arXiv:2401.09786_.
  * Kim et al. (2023) Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. 2023.  Llm4sgg: Large language model for weakly supervised scene graph generation.  _arXiv e-prints_ , pages arXiv–2310.
  * Lee et al. (2023) Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  In _International Conference on Machine Learning_ , pages 18893–18912. PMLR.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16_ , pages 121–137. Springer.
  * Liu et al. (2022a) Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2022a.  Deplot: One-shot visual language reasoning by plot-to-table translation.  _arXiv preprint arXiv:2212.10505_.
  * Liu et al. (2022b) Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2022b.  Matcha: Enhancing visual language pretraining with math reasoning and chart derendering.  _arXiv preprint arXiv:2212.09662_.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32.
  * Lu et al. (2023a) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023a.  Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models.  _arXiv e-prints_ , pages arXiv–2310.
  * Lu et al. (2023b) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023b.  Chameleon: Plug-and-play compositional reasoning with large language models.  _arXiv preprint arXiv:2304.09842_.
  * Luo et al. (2021) Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. 2021.  Chartocr: Data extraction from charts images via a deep hybrid framework.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 1917–1925.
  * Masry et al. (2023) Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023.  Unichart: A universal vision-language pretrained model for chart comprehension and reasoning.  _arXiv preprint arXiv:2305.14761_.
  * Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  _arXiv preprint arXiv:2203.10244_.
  * Meng et al. (2024) Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024.  Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning.  _arXiv preprint arXiv:2401.02384_.
  * Methani et al. (2020) Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020.  Plotqa: Reasoning over scientific plots.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1527–1536.
  * Poco and Heer (2017) Jorge Poco and Jeffrey Heer. 2017.  Reverse-engineering visualizations: Recovering visual encodings from chart images.  In _Computer graphics forum_ , volume 36, pages 353–363. Wiley Online Library.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Shao et al. (2023) Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023.  Prompting large language models with answer heuristics for knowledge-based visual question answering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14974–14983.
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Xia et al. (2023) Renqiu Xia, Bo Zhang, Haoyang Peng, Ning Liao, Peng Ye, Botian Shi, Junchi Yan, and Yu Qiao. 2023.  Structchart: Perception, structuring, reasoning for visual chart understanding.  _arXiv preprint arXiv:2309.11268_.
  * Xue et al. (2023) Wenyuan Xue, Dapeng Chen, Baosheng Yu, Yifei Chen, Sai Zhou, and Wei Peng. 2023.  Chartdetr: A multi-shape detection network for visual chart recognition.  _arXiv preprint arXiv:2308.07743_.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5579–5588.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020.  Unified vision-language pre-training for image captioning and vqa.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 13041–13049.
  * Zhou et al. (2023) Mingyang Zhou, Yi R Fung, Long Chen, Christopher Thomas, Heng Ji, and Shih-Fu Chang. 2023.  Enhanced chart understanding in vision and language task via cross-modal pre-training on plot table pairs.  _arXiv preprint arXiv:2305.18641_.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.

##  Appendix A Row-column Rendering

Fig. 8 presents a detailed exemple of our row-column rendering process. For a
simple chart and a original chart used in training stage, we extract rows and
columns from table and directly render them on to the charts since we have
ground-truth table in this stage. However, for original chart used in
inference stage, since we can not access the ground-truth table, rows and
columns are extracted from input chart and rendered onto the image with LMM
utilizing its powerful performance extracting textual information in the
image.

Figure 8: An example of row-column rendering for an input image in training
and inference stage. Figure 9: Overall prompt of Human-oriented chart
instruction.

##  Appendix B Human-oriented Prompt for Chart Reasoning

We present instructions in the prompt to mimic the way humans commonly
perceive charts. We name this approach Human-oriented chart instruction, which
is a simple yet effective methodology as shown in the QA results. The detailed
prompt is as shown in Fig. 9.

In the Human-oriented chart instruction, universal instructions applicable to
all chart types are provided first. It includes certain components that every
chart contains such as chart title, y-axis, x-axis, legend, and data labels,
facilitating the LMM to comprehend the chart. Furthermore, we guide the
alignment of charts and predicted tables by instructing the system within the
prompt to effectively comprehend the positions of columns and rows as well as
the outline of the chart, referring to the predicted table from Simplot.

Subsequently, instructions are described for each type of chart. When humans
commonly recognize a bar chart, the initial observation would be columns
distinguished by color. Following this observation, the legend within the
image is consulted to identify which color denotes which column. Therefore,
this intuitive recognition by humans has been incorporated into Bar Chart
Instruction 1 in Fig 9.

Upon identifying the columns, humans would proceed to observe which row each
bar corresponds to. Intuitively, for vertical bar graphs, the annotations
beneath the main chart are referred to, whereas for horizontal bar graphs,
annotations on the lateral sides are utilized to determine the rows. Bar Chart
Instruction 2 in Fig 9 reflects this role.

Humans then match the color of the bar with the annotated row, thereby
aligning each bar with its corresponding column and row. This process is
incorporated into Bar Chart Instruction 3 in Fig 9.

Lastly, humans would figure out the value corresponding to each row and column
by referencing the value annotated on the bar, the relative sizes between
bars, and the x-axis and y-axis. Notably, it is understood intuitively that
the value at the point where the bar terminates on the x-axis or y-axis
represents the value of that bar. This understanding is into Bar Chart
Instructions 4 and 5 in Fig 9, which is end of the instruction.

We generate instructions for line charts and pie charts in a similar manner as
described above. We argue that developing a language model that captures the
sequential and unconscious reasoning process involved in human chart
recognition through Human-oriented chart instruction not only mimics human
cognition effectively but also ensures the alignment of predicted tables with
chart images, leading to substantial performance enhancements at a lower cost.

##  Appendix C Details regarding Experiments

###  C.1 Dataset Description

In this paper, we evaluate our proposed method with widely-used dataset,
ChartQA (Masry et al., 2022). It consists of real-world charts containing
complex information, making it more aligned with our motivation compared to
other synthetic datasets such as PlotQA (Methani et al., 2020) and DVQA (Kafle
et al., 2018), both of which only include simple charts. Furthermore, ChartQA
comprises three types of charts (i.e., pie, bar, and line), whereas DVQA and
PlotQA comprise only one or two types of chart. Therefore, ChartQA is a more
challenging dataset and is desirable for evaluating performance on real-world
chart reasoning tasks.

###  C.2 Compared Methods

We use a wide range of models capable of performing chart reasoning as
baselines, categorizing them into three distinct categories.

The first category includes Vision-Language Pre-trained models (VLP) as
follows:

  * •

TaPas (Holmgren et al., 2012)

  * •

V-TaPas (Masry et al., 2022)

  * •

T5 (Bujard et al., 1987)

  * •

VL-T5 (Cho et al., 2021)

  * •

PaLI (Chen et al., 2022b)

  * •

Mini-GPT (Zhu et al., 2023)

  * •

LLaVa (Liu et al., 2023)

  * •

GPT-4V (Achiam et al., 2023)

The second category consists of supervised models, including followings:

  * •

ChartQA (Masry et al., 2022)

  * •

ChartT5 (Zhou et al., 2023)

  * •

Pix2Struct (Lee et al., 2023)

  * •

MatCha (Liu et al., 2022b)

  * •

Unichart (Masry et al., 2023)

  * •

ChartLlama (Han et al., 2023)

The final category is few-shot model, represented by our main baseline, Deplot
(Liu et al., 2022a). This model converts charts to tables similar to Simplot
and employs prompt-based methodologies such as Chain-of-Thought (CoT) (Wei et
al., 2022), Program-of-Thought (PoT) (Chen et al., 2022a) alongside LLM for
question reasoning.

###  C.3 Details regarding Proposed Metric

As illustrated on the right side of Fig. 10, columns generated by LMM such as
GPT-4 (e.g., "Percentage (%)" in the figure) often carry the same meaning as
the ground truth (e.g., "Value" in the figure) but have different Levenshtein
distances. Additionally, as shown on the table below of Fig. 11, in cases
where columns are not explicitly stated in the image, LMM without further
fine-tuning generates arbitrary columns. We are sure that, in the both cases,
there are absolutely no issues with chart reasoning using LMM. However, given
that $RMS$ is composed of the product of Levenshtein distance (from columns
and rows) and relative distance (from values), this leads to an apparent
degradation in performance of chart-to-table task even when there are no
issues with QA. Consequently, we construct a Levenshtein distance matrix by
concatenating column and row predictions with the ground truth, followed by
applying minimal cost matching to consider only the relative distance of each
pair. We adopt this Relative Distance ($RD$) as our main metric. This approach
allows us to accurately assess performance in value mapping.

Figure 10: An example of an extracted table that poses no issues for reasoning
despite having different column names from those in the ground-truth table.
Figure 11: An example where reasoning remains unaffected, yet the columns
extracted by LMM differ significantly due to the lack of clear column
description in the image.

We follow Deplot (Liu et al., 2022a) to define RD as the following equation,
with most formulas extracted from this paper. Following Liu et al. (2022a), we
use basic concept of Related Mapping Similarity ($RMS$), regarding tables as
unordered sets of mappings from row and column headers ($r,c$) to a single
value $v$. $p_{i}=(p^{r}_{i},p^{c}_{i},p^{v}_{i})$ and
$t_{j}=(t^{r}_{j},t^{c}_{j},t^{v}_{j})$ indicates each entity for the
predicted table $\mathcal{P}=\\{p_{i}\\}_{1\leq i\leq N}$ and the ground truth
table $\mathcal{T}=\\{t_{j}\\}_{1\leq j\leq M}$, respectively. Utilizing
Normalized Levenshtein Distance (${NL}_{\tau}$) (Biten et al., 2019), we
denote the distance between two keys $p_{i}$ and $t_{j}$ as
${NL}_{\tau}(p^{r}||p^{c},t^{r}||t^{c})$ where $||$ indicates concatenation of
string. The distance between values of table is computed with relative
distance $D_{\theta}(p^{v},t^{v})=min(1,||p^{v}-t^{v}||/||t^{v}||)$ and
distances larger than $\theta$ are set to the maximum of 1. Incorporating
these two distances, we obtain the similarity between two entities in a
mapping $D_{\tau,\theta}(p,t)$ as
$(1-{NL}_{\tau}(p^{r}||p^{c},t^{r}||t^{c}))(1-D_{\theta}(p^{v},t^{v}))$.

For $RMS$, Liu et al. (2022a) computes the pairwise similarity in
$\mathcal{P}$ and $\mathcal{T}$ using the cost function
$(1-{NL}_{\tau}(p^{r}||p^{c},t^{r}||t^{c}))$, obtaining a similarity matrix in
shape of $N\times M$ and identifying the minimal cost matching
$\mathbf{X}\in\mathbb{R}^{N\times M}$ between the keys (in the binary matrix
form).

While precision and recall is obtained from two mappings in Deplot Liu et al.
(2022a), we only use distance of the numeric entries to compute $RD$:

$RD_{precision}=1-\frac{\sum_{i=1}^{N}\sum_{j=1}^{M}\mathbf{X}_{ij}(D_{\theta}(p^{v},t^{v}))}{N},$
(5)

$RD_{recall}=1-\frac{\sum_{i=1}^{N}\sum_{j=1}^{M}\mathbf{X}_{ij}(D_{\theta}(p^{v},t^{v}))}{M}.$
(6)

Similar to Deplot, we compute ${RD}_{F1}$ as the harmonic mean of the
precision and recall.

##  Appendix D More Case Studies

Fig. 12 and 13 show that in both pie charts and bar charts, Deplot (Liu et
al., 2022a) misses relevant rows, generates inaccurate rows, thus fails to
create values, whereas Simplot consistently produces accurate tables. This
directly proves that our proposed method can perform well across various chart
types.

Furthermore, Fig. 14 presents how our Human-oriented chart instruction works
well immitating the way of human interpretation for charts. Without the
proposed prompt, LLM fails to detect exact location of the line component
"Favor", leading to a wrong answer. However, utilizing Human-oriented chart
instruction, LMM can locate the necessary component exactly, resulting in a
correct answer.

Figure 12: Case study. Extract all necessary rows in the pie chart. Figure 13:
Case study. Extract all necessary rows and precise value in the bar chart.
Figure 14: Case study. Interpreting charts similar to how humans do using
Human-oriented chart instruction. Figure 15: Error Analysis. Failure in row /
column rendering. Figure 16: Error Analysis. Failure in detecting a color in
the chart.

##  Appendix E Error Analysis

Here we present examples of Simplot failures, their reasons, and possible
solutions. In Fig. 15 (c), it can be observed that the extracted table from
the proposed method contains inaccurate rows. This is attributed to a line
break between the items "Exposure to false or incorrect information" and
"Losing the ability to Communicate face-to-face" in the input chart, leading
to the misinterpretation of each item as two separate entities during row
extraction. To address such issues, one possible solution would be to refine
the rows and columns by considering their contextual information before
rendering.

In addition, we find some examples where the LMM lacks specific abilities,
such as color detection. In Fig. 16, there is a question asking about the
color of "Jamaica" with the correct answer being orange, but the model
recognizes it as red and answers incorrectly. One possible solution would be
to broaden the range of similar colors for the model to answer together.


# Modeling Human Eye Movements with Neural Networks in a Maze-Solving Task

Jason Li, Nicholas Watters, Yingting (Sandy) Wang, Hansem Sohn, Mehrdad
Jazayeri
Department of Brain and Cognitive Sciences,
McGovern Institute for Brain Research
Massachusetts Institute of Technology
Cambridge, MA 02139
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

From smoothly pursuing moving objects to rapidly shifting gazes during visual
search, humans employ a wide variety of eye movement strategies in different
contexts. While eye movements provide a rich window into mental processes,
building generative models of eye movements is notoriously difficult, and to
date the computational objectives guiding eye movements remain largely a
mystery. In this work, we tackled these problems in the context of a canonical
spatial planning task, maze-solving. We collected eye movement data from human
subjects and built deep generative models of eye movements using a novel
differentiable architecture for gaze fixations and gaze shifts. We found that
human eye movements are best predicted by a model that is optimized not to
perform the task as efficiently as possible but instead to run an internal
simulation of an object traversing the maze. This not only provides a
generative model of eye movements in this task but also suggests a
computational theory for how humans solve the task, namely that humans use
mental simulation.

##  1 Introduction

Throughout the history of cognitive science, eye movements have been
appreciated as a window into the workings of the mind and brain Helmholtz
(1924); Liversedge Findlay (2000); Hayhoe Ballard (2005); König . (2016).
However, human eye movements are so rich and varied that characterizing them
is difficult even in simple tasks Land McLeod (2000); Beller. . (2022);
Gerstenberg . (2017). Building generative models of eye movements is an even
greater challenge Chen . (2017); Zoran . (2020), and to date most such work
focuses only on free-viewing or visual search contexts, not complex cognitive
tasks Kümmerer Bethge (2021); Zelinsky . (2020).

To tackle the problem of modeling task-driven saccade sequences, we designed a
maze-solving task. In this task, subjects must find the exit location of a
path in a maze given a starting point of the path (Figure 1). This task
provides an ideal platform for building generative models of eye movements
because it offers a near-limitless variety of spatial plans, yet eye movements
are largely consistent across humans Crowe . (2000), making them tractable to
model. Furthermore, this task may be solved using mental simulation of an
object traveling through the maze, so allows us to test mental simulation as a
computational theory guiding eye movements Gerstenberg . (2017); Ullman .
(2017); Ahuja Sheinberg (2019); Rajalingham . (2021).

In this work, we develop a novel general-purpose method for incorporating
features of human vision such as eccentricity-dependent visual acuity and
discrete saccades into a task-optimized, end-to-end differentiable recurrent
network. Using this method, we construct a space of models with and without
mental simulation constraints, and train these on the maze-solving task. We
collect eye movement data from human subjects playing the task and compare
this data to eye movements generated by the models to test multiple hypotheses
for how humans solve the task.

##  2 Related Work

Building generative models of human eye movements has been an active area of
research in psychology for decades Zelinsky . (2020); Kümmerer Bethge (2021);
Wedel . (2022). One approach to modeling eye movements is to hard-code the
heuristics of eye movements without employing task-driven learning. This
approach has seen some success in free viewing or visual search contexts Itti
. (1998); Zelinsky (2008); Zhang . (2005); Adeli . (2017); Zelinsky . (2013);
Eckstein (2011). However, our model differs from those approaches in that (i)
it learns a general policy for generating saccadic eye movements, so can in
principle be applied to any task, and (ii) is a neural network, hence can more
easily serve as a mechanistic model of the brain at an implementation level.

More recently, deep learning approaches to generate sequences of eye movements
have been developed, for example, by fitting a model directly to human data
Assens Reina . (2017); Sun . (2019); Xia . (2019); Yang . (2020); Kümmerer .
(2022). While these can provide impressive fits to human data, in this work
our goal differs in that we aim to build a model that shows emergent human-
like eye movements through task optimization, without explicitly fitting to
human eye movement data. Other deep learning models employ sequential
attention in general-purpose task-trained networks, but have not been
hypothesized as models of human eye movements or tested against human eye data
Gregor . (2015); Eslami . (2016); Adeli . (2022). In contrast, we develop our
models to generate human-like eye movements, collect (and open-source) human
eye data, and test our models against this data.

##  3 Methods

All of our data and code can be found at
https://github.com/jazlab/Maze_Task_2022, along with documentation and
instructions for replicating our results.

###  3.1 Task and maze dataset

In the maze-solving task, a subject is presented with a square maze and an
entrance point somewhere on its perimeter. This entrance point is one end of
an unbroken, non-branching path through the maze, which exits the maze at some
other uniformly sampled perimeter point. The subject is tasked to find this
exit point (see Appendix A for task instructions for human subjects). We
trained models on random mazes generated online. We also created a test set
for human and model comparison comprising 200 unique procedurally generated
mazes. See Appendix B for details about the maze generation algorithm.

Figure 1: Screenshots of the task presented to human subjects. At the start of
a trial, the subject fixates at a maze entrance point indicated by a white
cross (before maze presentation) or a green dot (after maze presentation).
Subjects must locate the correct exit point and press a button once they have
fixated their eye gaze on the exit point. At the button press, the true exit
position is indicated by another green dot (lower-left side in "Correct") and
a red dot ("Gaze") shows the reported position. The rightmost panel ("Error")
illustrates an error trial when subjects incorrectly identified the exit.

###  3.2 Human data collection

Fourteen human subjects volunteered to participate in the experiments after
providing informed consent. All participants (age: 18 to 65 years old, eight
female and six male) had normal or corrected-to-normal vision with no history
of neurological or psychiatric disorders. All experiments were approved by the
Committee on the Use of Humans as Experimental Subjects at the Massachusetts
Institute of Technology.

Subjects were seated in front of a LCD monitor (width: 53 cm, height: 30 cm;
Acer R240HY) at a distance of 66 cm. Each session started with a procedure for
calibrating eye positions using an optical eye tracker (EyeLink 1000 Plus, SR
Research). Eye position was monocularly sampled with 1 ms resolution while a
chin rest stabilized head of the participants. We monitored quality of the eye
signal during the experiment and repeated the eye calibration in the middle of
the session if needed. After receiving instruction for the maze-solving task
and several practice trials, data collection began. In each trial, we randomly
selected a maze from a predetermined test set. The test set included two
repetitions of each unique maze. Each participants completed one 1-hour
session, which consisted of approximately 400 trials. Maze stimuli (14 degree
of visual angle) and behavioral contingencies were controlled by an open-
source software (MWorks; mworks-project.org/) and Modular Object-Oriented
Games (MOOG) library Watters . (2021).

While solving the maze, humans’ eye movements were exclusively saccadic.
Consequently, we extracted saccades from the eye position data by first
filtering with a 4 ms Gaussian kernel and then thresholding eye velocity at 50
degrees of the visual angle per second. To prevent measurement noise of the
eye position from dominating metrics that we use to compare humans with
models, we recalibrated the raw eye position data for each trial. To do so, we
estimated a calibration error vector between the actual fixation point and the
gaze fixation point, and subtracted that vector from eye positions throughout
the trial. This recalibration did not affect our main findings.

###  3.3 Gaze recurrent neural network (RNN) models

We developed a task-optimized recurrent convolutional neural network model
that is equipped with a foveal module. The foveal module allows the model to
receive high acuity visual information near the fovea and low acuity
information in the periphery, like the human eye. The recurrent model is also
able to control the position of its fovea, allowing it to make eye movements.
The model is end-to-end differentiable, so can be trained via backpropagation,
through which an eye movement policy emerges from task-optimization.

We modeled the fovea by applying a circular exponential mask $e^{-d/\tau}$ to
the visual input, where $d$ is distance to the center of fovea and $\tau$ is a
scaling parameter. We chose 5 pixels as a value of $\tau$ (within a maze of 39
pixels), which is consistent with reported human peripheral visibility maps
Najemnik Geisler (2005); Strasburger . (2011). See Appendix C for results with
varying choices of $\tau$.

After applying the mask, we add independent noise to each pixel, sampled from
a normal distribution $\mathcal{N}(\mu=0,\,\sigma^{2}=0.05)$. This noise
washes out faint information in the peripheral tail of the foveal mask,
analogous to the decreased peripheral photoreceptor density in the human
retina. Note that this noise is essential to prevent the network from
exploiting the peripheral information available when only the mask is used.
Figure 2(a) shows a diagram of the model’s fovea mechanism. This foveal module
is general-purpose, and in theory can be incorporated into any RNN that takes
visual input. To our knowledge, this method is novel in the field.

(a) Differentiable fovea mechanism. Input is eye position and rendered maze;
output is noisy masked maze, the visual input for the next step of the gaze
RNN.

(b) Gaze RNN model. Memory CNN is a 3-layer Convolutional Neural Network.
Saccade CNN is a strided 3-layer CNN with two 3-layer MLP (Multi-Layer
Perceptron) heads for Cartesian eye position and ball position vectors. This
architecture can be unrolled through time for an arbitrary number of steps or
saccades.

Figure 2: Gaze RNN model diagram.

We implement three specific convolutional RNN models, exit, sim, and hybrid.
They all receive visual input via this foveal module, have the same internal
architecture, and can generate two outputs, the Cartesian coordinate for the
eye position (i.e., the center of fovea) and the Cartesian coordinate of the
next ball position (Figure 2(b)). The three models differ only in their
objective function.

####  3.3.1 Exit

The exit model is trained with a Mean Squared Error (MSE) loss between the eye
position at each step ($\hat{p}_{i}^{\text{eye}}$) and the maze exit point
($p^{\text{exit}}$), across all $n$ steps. There is no loss on the model’s
ball position output. This model represents an optimal exit-finding strategy
where the model moves its eyes to the exit in as few saccades as possible. We
minimize

$L_{\text{\sc{exit}}}=\frac{1}{n}\sum_{i=1}^{n}(\hat{p}_{i}^{\text{eye}}-p^{\text{exit}})^{2}$

####  3.3.2 Simulation

The sim model aims to predict the position of an imaginary ball that moves
from start to exit at constant velocity. To do so, we optimize this model with
a “simulation loss,” which is formulated as the MSE between each predicted
ball position ($\hat{p}_{i}^{\text{ball}}$) and the actual position of an
imaginary ball traveling at 10 pixels per time step ($p_{i}^{\text{ball}}$).
This ball speed was calculated from the human eye movement data as $l/n$
averaged over all trials, where $l$ is total maze length and $n$ is number of
saccades. In the sim model, eye position is not explicitly constrained, but
still plays a critical role in advancing the model’s visual field. We minimize

$L_{\text{\sc{sim}}}=\frac{1}{n}\sum_{i=1}^{n}(\hat{p}_{i}^{\text{ball}}-p_{i}^{\text{ball}})^{2}$

####  3.3.3 Hybrid

The hybrid model is trained with a weighted sum of the loss functions for the
exit and sim models. The model’s eye position must reach the exit quickly and
allow the model’s predicted ball positions to match the position of the
imaginary ball in the maze. The ratio of sim to exit loss weight is controlled
by a coefficient $\beta=\frac{1}{3}$, chosen so that the two loss terms have
similar magnitudes in a fully trained model. We minimize

$L_{\text{\sc{hybrid}}}=\beta\cdot L_{\text{\sc{exit}}}+(1-\beta)\cdot
L_{\text{\sc{sim}}}$

All models were trained on an NVIDIA GeForce GTX 1080 TI GPU with 4 GB RAM per
model and a total compute time of about 100 hours. Given computational
resource limitations, we trained one instance of each model, though see
Appendix C for results from additional instances in the context of
hyperparameter sweeps. Models were implemented in PyTorch Paszke . (2019) and
trained with 8 recurrent steps per maze, batch size 16, through 1.8 million
iterations using Adam optimizer Kingma Ba (2014) with learning rate 0.0003.
This was sufficient for each model’s loss to converge to a stable asymptote.
The training dataset was generated online using a custom procedural maze
generator with the same statistics (though not the same samples) as the test
set (see Appendix B for maze-generation details).

###  3.4 Baseline model

As a standard of comparison for the gaze RNNs, we created a baseline model
designed to match high-level human saccade statistics. This model iteratively
constructs saccade paths where the amplitude and angle of each saccade, as
well as the total number of saccades in the path, are sampled from the
corresponding distributions found in our human eye movement data (Figure 5).
For each trial, we construct 2,000 saccade paths and select the path whose
final fixation point is closest to the correct maze exit. For our maze test
set, this is sufficient to guarantee that the final fixation point falls
within 5 pixels of the maze exit greater than 90% of the time.

###  3.5 Metrics for comparing eye movement data

To quantify these results, we use two metrics for comparing eye movement
paths:

  * •

Nearest neighbors distance is computed as the mean of the nearest point in
path $A$ to each point $p_{B}$ in path $B$ and the nearest point in path $B$
to each point $p_{A}$ in path $A$:

$\displaystyle\mathcal{L}_{NN}$
$\displaystyle=\frac{1}{2}\cdot\left(\mathbb{E}_{p_{A}\in
A}\left[\min_{p_{B}\in B}||p_{B}-p_{A}||_{2}\right]+\mathbb{E}_{p_{B}\in
B}\left[\min_{p_{A}\in A}||p_{A}-p_{B}||_{2}\right]\right)$

  * •

Area between paths is computed as the total area of the polygon(s) formed
between paths $A$ and $B$. See Appendix D for details.

For both of these metrics, a lower value implies the paths $A$ and $B$ are
more similar.

##  4 Results

###  4.1 Saccade path similarity

Figure 3 shows the behavior of two representative human subjects, three RNN
models, and the baseline model for three example mazes in the test set.
Evidently, the saccadic eye movements in humans and models roughly follow the
correct path through the maze and successfully find the exit point. Humans
display a tendency to cut corners of the maze path. Qualitatively, out of the
gaze RNNs, the exit model seems most dissimilar to humans as it often makes
large saccades that are not present in human eye movements. The sim and hybrid
models make more uniform saccades that appear to better match human saccades.
The baseline model tends to generate erratic saccade paths that, by
construction, terminate near the exit point but do not resemble human saccade
paths.

Figure 3: Human and model behaviors on three sample mazes. Figure 4:  Metric
scores between model and human eye paths computed on the test set. To compute
these given a model and metric, for each test maze we compute the metric score
on each [model gaze path, subject gaze path] pair using all human subject
trials and 2 runs of a trained instance of the model on the test maze. We then
average all of these scores to obtain a total model-human similarity. Error
bars are 95% confidence intervals. Quantitatively, nearest-neighbors and area-
between-paths scores are: EXIT: [$0.987\pm 0.005$, $1.135\pm 0.012$]; SIM:
[$0.747\pm 0.007$, $0.679\pm 0.012$]; HYBRID: [$0.848\pm 0.006$, $0.811\pm
0.013$]; between-human mean: [$0.762$, $0.750$]. Note that the sim models
achieves better average similarity to humans than between-human similarity,
which is not impossible and implies that the model has lower variance than the
inter-subject variance.

Quantitatively, the sim model exhibits the lowest mean model-human distances
under both distance metrics (Figure 4). On the other hand, the exit model
produces the least human-like eye movement paths, comparable to those produced
by the baseline model. The hybrid model’s metric scores fall between those of
the other two gaze RNN models. Therefore, the sim model is the most human-like
of our three generative models.

###  4.2 Saccade vector similarity

Figure 5:  Saccade vector distributions for humans and models in visual angle.
For models, each plot shows 1,000 randomly sampled saccade vectors from a
dataset of 2 model evaluations on each test maze. The human plot shows 1,000
randomly sampled saccades from all human data aggregated across subjects.

In addition to comparing human and model saccade paths on a trial-by-trial
basis, we also compared their aggregate saccade vector distributions. Figure 5
shows a sample of the saccades executed by humans and each model on the test
set, with the tail of every saccade vector centered and the head plotted as a
point. In the human distribution, saccade angle is relatively uniform and most
saccade amplitudes are contained within a radius of 3 degrees. This is also
true for the baseline model’s distribution, which was sampled from the human
distribution. Consistent with the sample shown in Figure 3, the exit model’s
distribution shows many high-amplitude saccades, reflecting the large, erratic
saccades that model tends to favor. The sim model’s distribution is nearly
bounded by a square, which results from the right-angle maze geometry. Saccade
amplitude attains its maximum at the four cardinal directions because it is in
those directions that a constant-velocity ball can travel farthest through the
maze.

These results suggest that although the sim model’s eye movement paths most
closely resemble those of humans, it is not a complete model of humans’ eye
movement strategy when solving this task.

##  5 Limitations and Future Work

One limitation of our work is that the simulation model shows a more
constrained saccade vector distribution than humans (Figure 5), which may be a
consequence of its tendency to follow the path more faithfully than humans
(Figure 3). This close path-following with its fovea is an emergent property
of the model: The simulation loss was applied only to the model’s “ball
position” output, not its “eye position” output. Nonetheless, similarity to
human eye movements may be improved with variants of the SIM model, such as by
(i) varying the speed of the ball simulation, or (ii) introduce a non-constant
simulation speed, either learned by the model or computed based on
predetermined heuristics (such as speeding up on long straightaways and
slowing down near corners). Furthermore, exploration of the impact of
simulation speed on eye movements in the model may lead to predictions about
simulation speed in humans. Future work is needed to explore these
possibilities.

A second limitation of our work is that it is difficult to characterize the
biological plausibility of our foveal module’s hyperparameters. For instance,
the human visibility varies greatly based on context-dependent factors like
scene clutter, crowding, and luminance Pelli . (2007); Levi (2008);
Strasburger . (2011), so it is difficult to ascertain how our scaling
parameter $\tau$ compares to that of the human fovea. Nevertheless, we believe
that our choice of hyperparameters falls within reasonable bounds and when we
varied the hyperparameter $\tau$, the simulation model is still the most
similar to the human data (Appendix C).

A third limitation of our models is that they make one saccade for each RNN
timestep. This prevents them from capturing temporal aspects of human eye
movements, such as the duration of fixations between saccades, reaction times,
etc. Future work may address this by allowing the model to control fixation
durations, which could emerge from task training if longer fixations reduce
perceptual noise.

Finally, the mazes used to train and test the gaze RNN models had paths and
walls of the same width, while those presented to human subjects playing the
task had much thinner walls than paths. The resolution of the gaze RNN mazes
was limited by computational capacities. This may have resulted in humans
producing slightly different eye movement trajectories.

##  6 Conclusion

We find that in the maze-solving task, a gaze RNN trained to run an internal
simulation of a ball moving through a maze generates eye movements more
similar to those of human subjects than a model trained only to solve the task
as optimally as possible. This suggests that humans may employ a similar
mental simulation when performing this maze-solving task. Further work is
needed to explore the relationship between the biological plausibility of the
model fovea hyparparameters and model behavior. Future work also includes
incorporating our differential fovea method into RNNs trained on other tasks
to study the principles of human eye movements in domains beyond maze-solving.

## Acknowledgments and Disclosure of Funding

J.L. is supported by the MIT Quest for Intelligence. N.W. is supported by the
National Science Foundation. Y.W. is supported by the Simons Foundation. H.S.
is supported by a NARSAD young investigator grant from the Brain & Behavior
Research Foundation. M.J. is supported by the Simons Foundation, the McKnight
Foundation, and the McGovern Institute.

## References

  * Adeli . (2022) Adeli2022Adeli, H., Ahn, S. Zelinsky, GJ.  2022\.  A brain-inspired object-based attention network for multi-object recognition and visual reasoning A brain-inspired object-based attention network for multi-object recognition and visual reasoning.  bioRxiv.  https://www.biorxiv.org/content/early/2022/04/26/2022.04.02.486850 10.1101/2022.04.02.486850
  * Adeli . (2017) adeli2017Adeli, H., Vitu, F. Zelinsky, GJ.  2017\.  A model of the superior colliculus predicts fixation locations during scene viewing and visual search A model of the superior colliculus predicts fixation locations during scene viewing and visual search.  Journal of Neuroscience3761453–1467.
  * Ahuja Sheinberg (2019) ahuja2019behavioralAhuja, A. Sheinberg, DL.  2019\.  Behavioral and oculomotor evidence for visual simulation of object movement Behavioral and oculomotor evidence for visual simulation of object movement.  Journal of vision19613–13.
  * Assens Reina . (2017) assens2017Assens Reina, M., Giro-i Nieto, X., McGuinness, K. O’Connor, NE.  2017\.  Saltinet: Scan-path prediction on 360 degree images using saliency volumes Saltinet: Scan-path prediction on 360 degree images using saliency volumes.  Proceedings of the IEEE International Conference on Computer Vision Workshops Proceedings of the ieee international conference on computer vision workshops ( 2331–2338).
  * Beller. . (2022) Beller2022Beller., A., Xu, Y., Linderman, S. Gerstenberg, T.  2022\.  Looking into the past: Eye-tracking mental simulation in physical inference Looking into the past: Eye-tracking mental simulation in physical inference.  Cognitive Science Proceedings.
  * Chen . (2017) chen2017eccentricityChen, FX., Roig, G., Isik, L., Boix, X. Poggio, T.  2017\.  Eccentricity dependent deep neural networks: Modeling invariance in human vision Eccentricity dependent deep neural networks: Modeling invariance in human vision.  2017 AAAI Spring Symposium Series. 2017 aaai spring symposium series.
  * Crowe . (2000) crowe2000mentalCrowe, DA., Averbeck, BB., Chafee, MV., Anderson, JH. Georgopoulos, AP.  2000\.  Mental maze solving Mental maze solving.  Journal of Cognitive Neuroscience125813–827.
  * Eckstein (2011) Eckstein2011-poEckstein, MP.  201112\.  Visual search: a retrospective Visual search: a retrospective.  J. Vis.115.
  * Eslami . (2016) eslami_2016Eslami, SMA., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., kavukcuoglu, k. Hinton, GE.  2016\.  Attend, Infer, Repeat: Fast Scene Understanding with Generative Models Attend, infer, repeat: Fast scene understanding with generative models.  D. Lee, M. Sugiyama, U. Luxburg, I. Guyon R. Garnett (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 29).  Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2016/file/52947e0ade57a09e4a1386d08f17b656-Paper.pdf
  * Gerstenberg . (2017) Gerstenberg2017Gerstenberg, T., Peterson, MF., Goodman, ND., Lagnado, DA. Tenenbaum, JB.  2017\.  Eye-Tracking Causality Eye-tracking causality.  Psychological Science28121731-1744.  10.1177/0956797617713053
  * Gregor . (2015) gregor2015Gregor, K., Danihelka, I., Graves, A., Rezende, D. Wierstra, D.  2015\.  Draw: A recurrent neural network for image generation Draw: A recurrent neural network for image generation.  International conference on machine learning International conference on machine learning ( 1462–1471).
  * Hayhoe Ballard (2005) Hayhoe2005-jyHayhoe, M. Ballard, D.  200504\.  Eye movements in natural behavior Eye movements in natural behavior.  Trends Cogn. Sci.94188–194.
  * Helmholtz (1924) Helmholtz1924Helmholtz, HV.  1924\.  Helmholtz’s Treatise on Physiological Optics. Translated from the third German edition. Helmholtz’s treatise on physiological optics. translated from the third german edition.  The Hatton Press, Ltd.
  * Itti . (1998) itti1998Itti, L., Koch, C. Niebur, E.  1998\.  A model of saliency-based visual attention for rapid scene analysis A model of saliency-based visual attention for rapid scene analysis.  IEEE Transactions on pattern analysis and machine intelligence20111254–1259.
  * Kingma Ba (2014) kingma2014adamKingma, DP. Ba, J.  2014\.  Adam: A method for stochastic optimization Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * König . (2016) Konig2016-zaKönig, P., Wilming, N., Kietzmann, TC., Ossandón, JP., Onat, S., Ehinger, BV.Kaspar, K.  2016\.  Eye movements as a window to cognitive processes Eye movements as a window to cognitive processes.  J. Eye Mov. Res.951–16.
  * Kümmerer Bethge (2021) kummerer_2021Kümmerer, M. Bethge, M.  2021\.  State-of-the-Art in Human Scanpath Prediction State-of-the-art in human scanpath prediction.  CoRRabs/2102.12239.  https://arxiv.org/abs/2102.12239
  * Kümmerer . (2022) kummerer_2022Kümmerer, M., Bethge, M. Wallis, TSA.  202204\.  DeepGaze III: Modeling free-viewing human scanpaths with deep learning DeepGaze III: Modeling free-viewing human scanpaths with deep learning.  Journal of Vision2257-7.  https://doi.org/10.1167/jov.22.5.7 10.1167/jov.22.5.7
  * Land McLeod (2000) Land2000-wnLand, MF. McLeod, P.  200012\.  From eye movements to actions: how batsmen hit the ball From eye movements to actions: how batsmen hit the ball.  Nat. Neurosci.3121340–1345.
  * Levi (2008) Levi2008-twLevi, DM.  200802\.  Crowding–an essential bottleneck for object recognition: a mini-review Crowding–an essential bottleneck for object recognition: a mini-review.  Vision Res.485635–654.
  * Liversedge Findlay (2000) Liversedge2000Liversedge, SP. Findlay, JM.  2000\.  Saccadic eye movements and cognition Saccadic eye movements and cognition.  Trends in cognitive sciences46–14.
  * Najemnik Geisler (2005) Najemnik2005-heNajemnik, J. Geisler, WS.  200503\.  Optimal eye movement strategies in visual search Optimal eye movement strategies in visual search.  Nature4347031387–391.
  * Paszke . (2019) pytorch_2019Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G.Chintala, S.  2019\.  PyTorch: An Imperative Style, High-Performance Deep Learning Library Pytorch: An imperative style, high-performance deep learning library.  H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox R. Garnett (), Advances in Neural Information Processing Systems 32 Advances in neural information processing systems 32 ( 8024–8035).  Curran Associates, Inc.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Pelli . (2007) pelli2007crowdingPelli, DG., Tillman, KA., Freeman, J., Su, M., Berger, TD. Majaj, NJ.  2007\.  Crowding and eccentricity determine reading rate Crowding and eccentricity determine reading rate.  Journal of vision7220–20.
  * Rajalingham . (2021) Rajalingham2021Rajalingham, R., Piccato, A. Jazayeri, M.  2021\.  The role of mental simulation in primate physical inference abilities The role of mental simulation in primate physical inference abilities.  bioRxiv.
  * Strasburger . (2011) Strasburger2011-bvStrasburger, H., Rentschler, I. Jüttner, M.  201112\.  Peripheral vision and pattern recognition: a review Peripheral vision and pattern recognition: a review.  J. Vis.11513.
  * Sun . (2019) sun2019Sun, W., Chen, Z. Wu, F.  2019\.  Visual scanpath prediction using IOR-ROI recurrent mixture density network Visual scanpath prediction using ior-roi recurrent mixture density network.  IEEE transactions on pattern analysis and machine intelligence4362101–2118.
  * Ullman . (2017) ullman2017mindUllman, TD., Spelke, E., Battaglia, P. Tenenbaum, JB.  2017\.  Mind games: Game engines as an architecture for intuitive physics Mind games: Game engines as an architecture for intuitive physics.  Trends in cognitive sciences219649–665.
  * Watters . (2021) moog2021Watters, N., Tenenabum, J. Jazayeri, M.  2021\.  Modular Object-Oriented Games: A Task Framework for Reinforcement Learning, Psychology, and Neuroscience Modular object-oriented games: A task framework for reinforcement learning, psychology, and neuroscience.  arXiv preprint arXiv:2102.12616.  https://arxiv.org/abs/2102.12616
  * Wedel . (2022) wedel2022Wedel, M., Pieters, R. van der Lans, R.  2022\.  Modeling Eye Movements During Decision Making: A Review Modeling eye movements during decision making: A review.  Psychometrika1–33.
  * Xia . (2019) xia2019Xia, C., Han, J., Qi, F. Shi, G.  2019\.  Predicting human saccadic scanpaths based on iterative representation learning Predicting human saccadic scanpaths based on iterative representation learning.  IEEE Transactions on Image Processing2873502–3515.
  * Yang . (2020) yang2020Yang, Z., Huang, L., Chen, Y., Wei, Z., Ahn, S., Zelinsky, G.Hoai, M.  2020\.  Predicting goal-directed human attention using inverse reinforcement learning Predicting goal-directed human attention using inverse reinforcement learning.  Proceedings of the IEEE/CVF conference on computer vision and pattern recognition Proceedings of the ieee/cvf conference on computer vision and pattern recognition ( 193–202).
  * Zelinsky (2008) zelinsky2008Zelinsky, GJ.  2008\.  A theory of eye movements during target acquisition. A theory of eye movements during target acquisition.  Psychological review1154787.
  * Zelinsky . (2013) zelinsky2013Zelinsky, GJ., Adeli, H., Peng, Y. Samaras, D.  2013\.  Modelling eye movements in a categorical search task Modelling eye movements in a categorical search task.  Philosophical Transactions of the Royal Society B: Biological Sciences368162820130058.
  * Zelinsky . (2020) Zelinsky2020-izZelinsky, GJ., Chen, Y., Ahn, S. Adeli, H.  2020\.  Changing perspectives on goal-directed attention control: The past, present, and future of modeling fixations during visual search Changing perspectives on goal-directed attention control: The past, present, and future of modeling fixations during visual search.  Psychol. Learn. Motiv..
  * Zhang . (2005) zhang2005Zhang, W., Yang, H., Samaras, D. Zelinsky, G.  2005\.  A computational model of eye movements during object class detection A computational model of eye movements during object class detection.  Advances in neural information processing systems18.
  * Zoran . (2020) zoran2020towardsZoran, D., Chrzanowski, M., Huang, PS., Gowal, S., Mott, A. Kohli, P.  2020\.  Towards robust image classification using sequential attention models Towards robust image classification using sequential attention models.  Proceedings of the IEEE/CVF conference on computer vision and pattern recognition Proceedings of the ieee/cvf conference on computer vision and pattern recognition ( 9483–9492).

##  Appendix A Task instructions for human subjects

The goal of this “maze-solving” task is to identify the exit of the maze given
an initial entry position (indicated by a green ball). At the beginning of a
trial, you will see a white cross, which you are asked to look at. The
position of the cross corresponds to the initial entry position.

After a random delay, the maze and the green ball will appear on the screen.
From the entry where the green ball sits in, there will be one continuous path
until it hits one of the 4 boundaries of the maze.

After you identify the exit, fixate your eye gaze at the exit and then press
the left-arrow key to report the exit location (i.e., where you are looking
when pressing the key). As a feedback, the green ball will be revealed at the
correct location where it should exit the maze. Your reported exit location
will be shown with a red ball. Therefore, if both green and red ball are close
or right next to each other, that means your response was correct.

Note that we will record your eye gaze throughout the trial and so do your
best not to move your head during the experiment.

Every 50 trials, a gray screen will appear and you are given a break as long
as you want. When you are ready to resume the task, press the left-arrow key
again. In total, you will complete 400 trials.

##  Appendix B Maze Generation

We procedurally generated maze via a simple layering procedure. First, we
implemented a path-generation algorithm, which sampled a random path within
the $20\times 20$ maze grid by picking a random starting edgepoint and taking
a random walk with turn probability $0.2$ and minimum inter-turn distance $3$
until reaching an edgepoint. Then, to generate a maze we layered such randomly
generated paths with occlusion until every grid loation in the maze was
covered by some path.

The code for this algorithm can be found in our open-sourced repo,
https://github.com/jazlab/Maze_Task_2022.

##  Appendix C Fovea Size Sweeping

$\tau$ (pixels) | EXIT | SIM | HYBRID
---|---|---|---
8 | 5.04 | 3.10 | 3.71
5 | 4.65 | 3.11 | 3.85
3.33 | 4.36 | 3.14 | 3.15
(a) Nearest neighbors

$\tau$ (pixels) | EXIT | SIM | HYBRID
---|---|---|---
8 | 294.3 | 134.4 | 202.7
5 | 270.0 | 137.9 | 191.7
3.33 | 242.8 | 139.2 | 142.6
(b) Area between paths

Table 1: Similarity of gaze RNNs to humans across three values of $\tau$.

##  Appendix D Area between paths metric

The area-between-paths metric measures similarity of two paths in space. It is
computed as the total plane area of all polygon(s) formed between the two
paths. See Figure 6 for an illustration.

Figure 6: Illustration of area between paths. Given paths A and B, the area
between them is shaded gray.