ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding
Under review as a conference paper at ICLR 2025

CHARTMOE: MIXTURE OF DIVERSELY ALIGNED EX-
PERT CONNECTOR FOR CHART UNDERSTANDING

Anonymous authors
Paper under double-blind review

ABSTRACT

Automatic chart understanding is crucial for content comprehension and docu-
ment parsing. Multimodal Large Language Models (MLLMs) have demonstrated
remarkable capabilities in chart understanding through domain-specific alignment
and fine-tuning. However, current MLLMs still struggle to provide faithful data
and reliable analysis only based on charts. To address it, we propose ChartMoE,
which employs the Mixture of Expert (MoE) architecture to replace the tradi-
tional linear projector to bridge the modality gap. Specifically, we train several
linear connectors through distinct alignment tasks, which are utilized as the foun-
dational initialization parameters for different experts. Additionally, we introduce
ChartMoE-Align, a dataset with nearly 1 million chart-table-JSON-code quadru-
ples to conduct three alignment tasks (chart-table/JSON/code). Combined with
the vanilla connector, we initialize different experts diversely and adopt high-
quality knowledge learning to further refine the MoE connector and LLM param-
eters. Extensive experiments demonstrate the effectiveness of the MoE connector
and our initialization strategy, e.g., ChartMoE improves the accuracy of the previ-
ous state-of-the-art from 80.48% to 84.64% on the ChartQA benchmark.

1

INTRODUCTION

Charts serve as a fundamental tool for data visualization, with automated chart interpretation gain-
ing prominence in domains such as text analysis Hoque et al. (2017), scientific research Hsu et al.
(2021), and policy-making Wu et al. (2024). Chart understanding is a complex task that demands
the identification of visual cues, the comprehension of intricate interactions, and the precise infer-
ence of values informed by prior knowledge. Previous work Lee et al. (2023); Liu et al. (2023b;a)
typically pre-trained on domain-specific charts, which are constrained by limited resources and nar-
row task focus. In contrast, Multi-modal Large Language Models (MLLMs) Li et al. (2023); Liu
et al. (2023d); Bai et al. (2023a); Ye et al. (2023b); Chen et al. (2023a); OpenAI (2023) exhibit sub-
stantial potential in image comprehension and instruction following. The community has achieved
advanced progress by creating chart understanding datasets Liu et al. (2023c); Han et al. (2023);
Masry et al. (2024b); Xu et al. (2023) and applying supervised fine-tuning based on well-performed
MLLMs Meng et al. (2024); Yan et al. (2024); Carbune et al. (2024). With the exponential growth
of chart data, automated chart interpretation via MLLMs is emerging as a promising avenue.

Recent studies advocate for chart alignment as a foundational step for LLaVA-like MLLMs Liu
et al. (2023d); Zhang et al. (2023); Xue et al. (2024), which bridge the visual encoder and LLM
through a linear connector. They usually utilize the chart-to-table alignment task to train the linear
connector effectively Meng et al. (2024); Yan et al. (2024); Hu et al. (2024). However, tables only
provide basic information, such as numerical values and titles, which fail to capture the full range of
chart elements. Despite some efforts to align with more informative text Yan et al. (2024), the heavy
alignment tasks may lead to the erosion of the connector’s general capabilities, e.g., instruction fol-
lowing and visual counting, which are derived from the pre-training on large-scale visual-language
data. To mitigate knowledge forgetting, one intuitive approach is to further tune the connector with
its original data, which results in redundant training and computational burden.

In this paper, we try to address these challenges via Mixture of Experts (MoE) architecture Zoph
et al. (2022). MoE enhances model capacity by activating a subset of experts through a routing
network. Since the alignment tasks work on the connector, we replace only the linear projector

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Overview and capabilities of ChartMoE: We introduce a MoE architecture connector and provide
visualizations of the top-1 expert selection (refer to Fig. 6 and Appendix B for details). ChartMoE can extract
highly precise values and provide flexible chart editing through code-based interactions.

layer with MoE while keeping the vision encoder and LLM frozen. Our insight lies in the expert
initialization manner. Random initialization can lead to training instability and convergence at sub-
optimal points (Fig. 4). Recent co-upcycling initialization Komatsuzaki et al. (2023) addresses this
issue by duplicating the vanilla connector parameters across all experts. However, it fails to avoid
the dilemma of expert homogenization, where the experts end up with similar functionalities.

In contrast, we attempt to inject distinct prior knowledge into each expert first to tackle these chal-
lenges. Unlike natural images, charts can be represented in various text formats, e.g., tables, at-
tribute JSON, and rendering code. As shown in Fig. 1& 2, in addition to chart-table, we introduce
chart-JSON alignment to capture detailed elements like color or topological relationships and chart-
code alignment to incorporate rendering details such as numerical values and color hex codes (refer
to Appendix C). We independently conduct various alignment tasks to capture more diverse chart
features and thus obtain three distinct initialization approaches. We also retain the vanilla connector
to effectively preserve the capabilities of the MLLM on general tasks.

Building upon the proposed four initialization manners, we introduce ChartMoE, an SFT-based
MLLM with the MoE connector for chart comprehension and reasoning. Interestingly, we observe
that experts in ChartMoE exhibit distinct visual token preferences, e.g., the vanilla expert favors
background tokens while other experts focus more on tokens with legends or numbers (Fig. 6 and
Appendix B). Considering that the distribution of visual tokens is naturally imbalanced in chart sce-
narios, we remove the expert balanced loss in MoE and obtain further performance gain. Due to the
scarcity of rich text for chart alignment, we design a pipeline (Fig. 3) to generate nearly 1 million
quadruplets chart-table-JSON-code to build the ChartMoE-Align dataset for alignment. We train
ChartMoE in 3 stages. First, we initialize experts via the proposed four manners. Then, we conduct
high-quality knowledge learning using the MMC instruction Liu et al. (2023c) to train the rout-
ing network, expert connectors, and LoRA Hu et al. (2022) modules. Finally, we employ annealing
training on ChartQA Masry et al. (2022) and ChartGemma Masry et al. (2024b). We further integrate
the Program of Thought (PoT) prompting Chen et al. (2023b) to enhance mathematical capabilities.
ChartMoE can be deployed on a single A100-40G GPU and achieves state-of-the-art (SOTA) per-
formance. ChartMoE provides more precise numbers and comprehensive attributes when queried
with charts (Appendix E). In summary, our contributions are:

a) We present ChartMoE for faithful and reasonable chart understanding, with the connector based
on Mixture of Expert architecture, to bridge the chart and LLM branches. All experts are initial-
ized based on various alignment training tasks to avoid expert homogenization.

b) We introduce ChartMoE-Align, a large-scale dataset with nearly 1 million meticulous chart-

table-JSON-code quadruplets for chart alignment pre-training.

c) We propose to train ChartMoE with a three-stage training paradigm, including connector align-

ment pre-training, high-quality knowledge learning, and annealing chart tuning.

d) Extensive quantitative and qualitative studies demonstrate that ChartMoE significantly outper-

forms previous state-of-the-art across several benchmarks by a large margin.

2

LLMViTChartTextAnswerLabelGateRedraw the chart with python matplotlib code output. Pay attention to the line color!import pandas as pdimport matplotlib.pyplot as pltplt.style.use("classic")data = {'China': [100, 150, 200, 250, 300, 350, 400, 450],'USA': [50, 75, 100, 125, 150, 175, 200, 225],'India': [25, 40, 55, 70, 85, 100, 125, 150],'Japan': [15, 20, 25, 30, 35, 40, 45, 50],'Germany': [10, 12, 14, 16, 18, 20, 23, 24]}years = ['2015', '2016', '2017', '2018', '2019', '2020', '2021', '2022']df = pd.DataFrame(data, index=years)plt.plot(df['China'], label='China', linewidth=1, marker='o', color='#0000FF') # blueplt.plot(df['USA'], label='USA', linewidth=1, marker='o', color='#FF0000') # redplt.plot(df['India'], label='India', linewidth=1, marker='o', color='#008000') # greenplt.plot(df['Japan'], label='Japan', linewidth=1, marker='o', color='#808080') # greyplt.plot(df['Germany'], label='Germany', linewidth=1, marker='o', color='#FFFF00')# yellowfor country in df.columns:    for i, value in enumerate(df[country]):       plt.text(i, value + 2, f'{value}', ha='center',                    va='bottom', fontsize=8)plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))plt.xlabel('Years')plt.ylabel('Petabytes per month')plt.title('Mobile data usage by country')plt.tight_layout(rect=[0, 0, 0.85, 1])plt.show()  Redraw the chart with python matplotlib code output. 1) Pay attention to the line color! 2) Change the line marker to star!...
# Draw the chart, set the line width and mark pointsplt.plot(df['China'], label='China', linewidth=1, marker='*', color='#0000FF') # blueplt.plot(df['USA'], label='USA', linewidth=1, marker='*', color='#FF0000') # redplt.plot(df['India'], label='India', linewidth=1, marker='*', color='#008000') # greenplt.plot(df['Japan'], label='Japan', linewidth=1, marker='*', color='#808080') # greyplt.plot(df['Germany'], label='Germany', linewidth=1, marker='*', color='#FFFF00')# yellow...Redraw the chart with python matplotlib code output.  1) Pay attention to the line color! 2) Circle the maximum value in red!...
# get max valuemax_value = df.max().max()max_index = df.stack().idxmax()country_name = max_index[1]year_index = max_index[0]# Circle max value in redplt.scatter(year_index, max_value, s=200, edgecolor='red', facecolor='none', linewidth=2)...
Query ChartToken-wise Top-1 Expert Selection in MoE ConnectorPlot the US data as a percentage pie chart in python.import pandas as pdimport matplotlib.pyplot as pltdata = {'China': [100, 150, 200, 250, 300, 350, 400, 450],'USA': [50, 75, 100, 125, 150, 175, 200, 225],'India': [25, 40, 55, 70, 85, 100, 125, 150],'Japan': [15, 20, 25, 30, 35, 40, 45, 50],'Germany': [10, 12, 14, 16, 18, 20, 23, 24]}years = ['2015', '2016', '2017', '2018', '2019', '2020', '2021', '2022']df = pd.DataFrame(data)usa_data = df['USA']plt.figure(figsize=(8, 8))plt.pie(usa_data, labels=years, autopct='%1.1f%%', startangle=140)plt.title('Percentage Distribution of Mobile Data Usage (USA)')plt.show()According to this chart, how much mobile data  does China use in 2020?It is arond 350 Petabytes per month.Chart QAReplotEidtingHighlightingTransformationUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

2 RELATED WORK

Multimodal large language models leverages a connector to bridge the gap between large lan-
guage models Touvron et al. (2023); Radford et al. (2018); Brown et al. (2020); Zhang et al. (2022);
Zheng et al. (2023) and vision encoders Radford et al. (2021); Oquab et al. (2023) to enable en-
riched capabilities of comprehension and instruction following. Approaches such as BLIP2 Li et al.
(2023), Flamingo Alayrac et al. (2022), mPLUG-Owl Ye et al. (2023b), and Qwen-VL Bai et al.
(2023b) utilize QFormers or Resamplers to align modalities on extensive datasets of image-text
pairs. LLaVA Liu et al. (2023d; 2024b) is the pioneering work to extend the instruction tuning
paradigm to visual tasks with text-only GPT-4 OpenAI (2023), achieving tremendous performance
using a simple MLP without compromising visual information to refine the multimodal alignment.
Some works Lin et al. (2023); Tong et al. (2024b;a) explore the combination of various vision
encoders, complementarily enhancing visual representations to bolster the fine-grained visual per-
ception of MLLMs. Despite efforts in structural design, training strategies and data quality remain
crucial in the advancement of MLLMs.

Chart Reasoning refers to chart analysis, summarization, and etc. Existing methods can be cate-
gorized as 1) Two-stage methods use specialized extraction modules to generate intermediary rep-
resentations of chart information, like tables, which are provided as textual prompts for LLMs.
Pix2Struct Lee et al. (2023) aligns markdown data with charts. MatCha Liu et al. (2023b) aligns
various data formats (e.g., tables and code) with charts on several downstream tasks. DePlot Liu
et al. (2023a) fine-tunes Pix2Struct for table extraction and uses LLMs to process queries based on
the extracted data. ChartVLM Xia et al. (2024) employs a discriminator to ascertain the necessity
of intervention by LLMs for a given query. 2) End-to-end methods strive to tackle chart reasoning
challenges with a unified model. ChartLlama Han et al. (2023) incorporates diverse charts and down-
stream tasks based on LLaVA Liu et al. (2023d). ChartPaLI Carbune et al. (2024), ChartAst Meng
et al. (2024), and MMC Liu et al. (2023c) conduct alignment on table-chart pairs. UReader Ye
et al. (2023a) aligns all data with markdown, while mPLUG-Owl2 Ye et al. (2023c) achieves supe-
rior performance with high-resolution inputs. ChartThinker Liu et al. (2024c) and DOMINO Wang
et al. (2023) propose the CoT Wei et al. (2022) for chart reasoning. LaMenDa Zhuowan et al.
(2024) trains MLLMs via step-by-step reasoning QA. ChartReformer Yan et al. (2024) introduces
chart-JSON alignment, while OneChart Chen et al. (2024) aligns charts with Python dictionaries.
MiniGPT-v2 Chen et al. (2023a), Doc-Owl Hu et al. (2024), and TinyChart Zhang et al. (2024)
tackle the reasoning efficiency for high-resolution charts by merging tokens.

3 CHARTMOE

3.1 ARCHITECTURE

The ChartMoE is based on InternlmXC-v2 Dong et al. (2024) due to the concise LLaVA-like archi-
tecture Liu et al. (2023d) and performance on par with GPT-4 on text-image comprehension. The
base model includes a vision encoder and a LLM connected by a two-layer MLP. ChartMoE replaces
the MLP with a MoE architecture as the connector to leverage diverse prior knowledge.

Vision Encoder. We utilize CLIP ViT-Large Radford et al. (2021) as the vision encoder, leveraging
its rich prior knowledge gained from training on millions of image-text pairs. Considering the
impact of chart resolution on performance, we set the input resolution to 490 × 490 to strike a
balance between efficiency and performance. Formally, the visual encoder MV (·) will project the
chart I into N tokens V := {v1, v2, . . . , vN }, where N = 1225 in the ChartMoE.

Mixture-of-Experts Connector. As illustrated in Fig. 2c, the MoE architecture employs a parallel
multi-expert collaboration approach. This architecture comprises L experts ME(·), each designed
with the same linear layer as the baseline. For a visual token vi given by MV , the gating network
MG(·) will calculate the routing weight gj(vi) of each expert ME
j (·) and select top-K to activate.
Finally, the tokens processed by each expert ME
j will be averaged according to the weight gj(vi)
given by MG to get the token ˆvi for the LLM branch ML.

Large Language Model. Following the baseline, we employ the InternLM2-7B-ChatSFT variant
as the LLM ML, implemented as a transformer decoder with a causal attention mask. We concate

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 2: Overview of proposed ChartMoE. (a) Examples of alignment instructions. (b) We conduct three
different alignment tasks in parallel. (c) We initialize MoE connectors in four different manners and train the
gate network, experts, and LoRA during the supervised fine-tuning stage.

the visual tokens ˆV := {ˆv1, ˆv2, . . . , ˆvN } given by MoE connector with the M input text T tokens
T := {t1, t2, . . . , tM } to form the input token sequence for the LLM ML. Formally, give the chart
I and instruction T , the output O of proposed ChartMoE can be formulated as:

{v1, v2, . . . , vN } = MV (I),

L
(cid:88)

ˆvi =

gj(vi)ME

j (vi),

g(vi) = Top(σ(MG(vi)); K),

j
O = ML({ˆv1, ˆv2, . . . , ˆvN ; t1, t2, . . . , tM }),

(1)

(2)

(3)

where σ indicates softmax and the Top(·; K) will reset the non-Top K routing weight to 0.

Initialization of Expert. Previous approaches initialize expert parameters via 1) Random initializa-
tion, which may lead to convergence difficulties during supervised fine-tuning, and 2) Co-upcycling
initialization Komatsuzaki et al. (2023), i.e., copy baseline connector parameters to each expert,
which may lead to homogenization of experts. ChartMoE proposes initializing experts’ parameters
through distinct alignment tasks. We eliminate the load-balancing loss typically used in standard
MoE architectures to equalize expert activation frequencies, as our initialization approach allows
each expert to specialize in its preferred visual tokens, which inherently exhibit biased distributions.

3.2 ALIGNMENT PRE-TRAINING.

The key insight of ChartMoE is the experts’ initialization parameters from the different alignment
pre-training (Fig. 2a). Specifically, as illustrated in Fig. 2b, we align expert connectors using three
distinct alignment tasks, where only the connector parameters will be updated. We visualize the
visual token preferences of each expert for both chart (Fig. 6& 12) and non-chart (Fig. 11) images.

Alignment with Table. Charts convey key information that can be more precisely expressed in tab-
ular form, and LLMs are particularly adept at processing such structured data. Hence, we introduce
a chart-table alignment task, aiming to translate chart content into tabular format. The connector
is trained to convert chart information into corresponding CSV tables, thereby improving model
performance in numerical data extraction and chart interpretation.

Alignment with JSON. Although tables capture the numerical information from charts, they miss
semantic details such as colors, shapes, and fonts. To fill this gap, we propose a chart-JSON align-
ment task, which represents chart attributes in JSON format. This task requires the connector to
focus not only on the numerical data but also on visual and semantic properties. Accurately extract-
ing chart attributes is essential for tasks like chart redrawing and editing.

Alignment with Code. To fully align with charts, we further introduce a chart-code alignment
task. Since the underlying drawing code fully defines a chart, this approach enables the connector to
convert the chart’s visual tokens into representations in the LLM domain. Notably, we provide the

4

Translate the chart into csv format.Age group, Nov 2010, Dec 2011, Jan 2012--- , --- , --- , ---18-29 , 6% , 7% , 18% 30-49 , 5% , 12% , 24%50-64 , 9% , 11% , 19%65 and older , 4% , 8% , 12%CSVTranslate the chart into JSON with all available attributions.{    "type_agnostic": {        "x_font_name": "sans-serif",        "x_font_size": "medium",        "y_font_name": "monospace",        "y_font_size": "x-large",     ...}JSONConvert this chart to python style code.import matplotlib.pyplot as pltimport numpy as np# vis toolplt.style.use('default')# datax = ['Age group', 'Nov 2010', 'Dec 2011', 'Jan 2012']y = [[6, 7, 18], [5, 12, 24], [9, 11, 19], [4, 8, 12]]...Code(a) Alignment InstructionsVisual TokensText TokensViT EncoderAlignment InstructionsChartConnector× N LayerLLM(b) Alignment PretrainingVisual TokensE0E1E2LLMVisual TokensText TokensChartLoRArInstructionsViT EncoderTop-kGateE0E1E2E3VanillaConnectorWeighted  Sum× N Layer(c) Supervised FinetuningVisual TokensUnder review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 3: Overview of ChartMoE-Align data generation pipeline. The charts are plotted by Python matplotlib.

drawing code explicitly, including precise numerical values and rendering attributes, e.g., numbers
represented in Python lists and colors in hexadecimal code. Refer to Fig. 13 for more detailed cases.
The code enables the model to perform in-depth summarization, analysis, and editing of charts. This
expert is significantly more sensitive to the trends and key elements in the charts.

ChartMoE-Align generation pipeline. As Fig. 3 illustrates, 1) We filter charts with meta CSV
from existing datasets Masry et al. (2022); Methani et al. (2020) and data generated by LLMs Chen
et al. (2024). 2) We use a fine-tuned Deplot Liu et al. (2023a) to inverse the plotting attributes
following the templates provided by ChartReformer and randomly sample missing attributes from
the predefined set. 3) We create code templates for different types of charts and generate plotting
code based on the meta CSV and extracted JSON attributes. Note that all values and attributes in the
code are explicitly represented. 4) We retain the (table, JSON, code, chart) quadruples
that pass compilation. Tab. 1 shows the data sources & size and refer to Appendix C for details.

3.3 SUPERVISED FINE-TUNING.

We initialize ChartMoE using the structure shown in Fig. 2c after aligning the connectors across 3
distinct tasks separately. We also retain the vanilla connector to maintain the baseline’s excellent
dialogue capabilities, which aligns with the principle of residual optimization He et al. (2016). We
train the MoE connector and LLM during this stage with LoRA Hu et al. (2022), as shown in Fig. 2c.
Considering the training principles proposed in LLaVA-NeXT Liu et al. (2024a), this stage is divided
into high-quality knowledge learning and chart-specific annealing training.

High-Quality Knowledge Learning. We adopt MMC Liu et al. (2023c) to enhance the ChartMoE’s
knowledge. MMC includes a variety of chart types and tasks such as chart-related question answer-
ing, translation, extraction, reasoning, and analysis. Considering data quality, we only utilize the
MMC-Instruction subset, which has been manually verified. Notice that the quality of instruction
data is more important than quantity in this stage.

Chart Specific Annealing Tuning. Following Llama-v3.1 Team et al. (2024b), we perform anneal-
ing tuning before evaluating mainstream benchmarks. We increase the learning rate and conduct
instruction tuning using the training sets of ChartQA and ChartGemma to adjust the query styles
and answer formats of these benchmarks.

Program of Thought (PoT) Inference. We require the model to generate the variables and opera-
tion code rather than producing direct answers. This inference pipeline addresses the mathematical
capabilities by employing Python to handle the logical computations, which is the shortcoming of
all open-sourced models. With better numerical extraction abilities, PoT can significantly enhance
our ChartMoE’s question-answering performance.

4 EXPERIMENT

4.1

IMPLEMENTATION DETAILS

During the alignment stage, we train the connector parameters and keep the visual encoder and LLM
parameters fixed for 1 epoch. The learning rate is set to 5e-5 with a warmup phase covering the first
1% of training steps. In the supervised fine-tuning stage, we continue training the connector while
employing LoRA to update the LLM parameters with the rank of 64. The learning rate is adjusted to
1e-5 for the high-quality knowledge learning period and 5e-5 for the chart-specific annealing tuning
period. The weight decay is 0.1 for all stages. We use the cosine annealing learning rate schedule.
The global batch size is set to 64 for all stages. The training process is conducted on A100-40G
GPU, with the alignment stage taking approximately 240 GPU hours, the knowledge learning stage
taking 138 GPU hours, and the chart-specific annealing tuning stage taking around 76 GPU hours.

5

CSV TableFilter out meta CSV data from existing datasets Generate meta CSV data by large language modelsAttribution JSONRandom sample from predefined attribution setsInversion from chart by finetuned Deplot modelChartExecute renderGet executable code with predefined templates and json propertiesPython CodeFailedSuccessRenderKeep the quadrupleDiscard the quadrupleUnder review as a conference paper at ICLR 2025

4.2 EVALUATION METRICS

ChartQA Masry et al. (2022) test split consists of
1,250 questions in both human and augmented parts.
The charts are three common chart types and are
sourced from the real world.
It features a variety
of human-crafted questions and answers to evaluate
models’ understanding, reasoning, and data extrac-
tion skills. ChartQA adopts relaxed accuracy, which
is highlighted shortcomings by recent studies Chen
et al. (2024); Xu et al. (2023), such as simplistic
string matching and direct float conversion. There-
fore, we improve it by 1) using regular expression
matching to extract number values, 2) optimizing
string matching for short answers, and 3) demon-
strating model performance under various relaxed
margins. We adopt it for all experiment results.

Table 1: Datasets used for training ChartMoE.
We conduct alignment pre-training with synthetic
data and supervised tuning with high-quality, real-
world data. Only ChartQA is used in the ablation
due to GPU constraints.

Source

Data Type

Task Type

Samples

Alignment Training

ChartQA

synthetic

PlotQA

synthetic

ChartY

synthetic

chart to table
chart to JSON
chart to code
chart to table
chart to JSON
chart to code
chart to table
chart to JSON
chart to code

18.3K
18.3K
18.3K

157K
157K
157K

763.6K
763.6K
763.6K

Total
2.8M
Usage: Table = 500K JSON = 200K Code = 100K 800K

ChartBench Xu et al. (2023) focuses on charts with-
out data point annotations.
It includes a broader
range of chart types, with 9 main categories and
42 subcategories, each containing 50 charts. Chart-
Bench focuses on extracting numerical values, pos-
ing a greater challenge as models cannot depend on
OCR for precise answers. It adopts Acc+ for judg-
ments and relaxed accuracy for NQA tasks. The
benchmark proposes to extract number values by LLMs first, which is omitted for the stratifying
instruction-following ability of ChartMoE.

High-Quality Knowledge Learning
QA & reasoning
& summariztion

QA
QA & PoT & reasoning
& summariztion

Chart Specific Annealing Tuning
real world

synthetic
& real world

ChartGemma

real world

28.3K×2

ChartQA

220.8K

163.2K

MMC

410K

Total

ChartFC Akhtar et al. (2023a) & ChartCheck Akhtar et al. (2023b) adopt accuracy to verify
whether the claim aligns with the input chart, marking a significant advancement in chart recog-
nition and reasoning abilities. This identifies the potential hallucinations in chart-related contexts.
The ChartFC test set has 1,591 questions, and the ChartCheck test set has two splits, containing 937
questions and 981 questions.

4.3 COMPARATIVE MODELS

General MLLMs. We compare PaliGemma Beyer et al. (2024), LLaVA-v1.5 Liu et al. (2023d) with
an MLP connector, Qwen-VL Bai et al. (2023b) with a Qformer Li et al. (2023) connector, DocOwl-
v1.5 Hu et al. (2024) that employs multi-level image resolution and token convolution techniques,
and the current open-source SOTA, InternlmXC-v2 Dong et al. (2024).

Specialist Chart Models. Previous works specifically design models and algorithms for chart ques-
tion answering. We compare Pix2Struct Lee et al. (2023), Matcha Liu et al. (2023b), UniChart Masry
et al. (2023), and Deplot Liu et al. (2023a). Notably, Deplot fails to handle questions in arbitrary
formats, so we extract table information with Deplot and use LLaVA-v1.6 to answer the questions.

Chart MLLMs. Chart-oriented MLLMs are the promising direction for utilizing prior knowledge
of LLMs. ChartLLaMA Han et al. (2023) proposes to generate high-quality instruction data to
improve chart question-answering capabilities. ChartAst Meng et al. (2024) suggests aligning the
connector with chart-table pairs before supervised fine-tuning. ChartVLM Xia et al. (2024) uses
different decoders to handle different questions based on their difficulty. ChartInstruct Masry et al.
(2024a) conducts large-scale chart instruction tuning based on general MLLMs. OneChart Chen
et al. (2024) converts the chart to the table with a dedicated decoder and uses LLMs to answer ques-
tions. ChartGemma Masry et al. (2024b) proposes more instruction data and achieves efficient chart
reasoning based on SigLIP Zhai et al. (2023) and Gemma-2B Team et al. (2024a). TinyChart Zhang
et al. (2024) adopts token merge to reduce visual tokens and enable high-resolution chart input.

4.4 MAIN RESULTS

ChartQA. Tab.2 presents detailed comparisons of ChartMoE on ChartQA. ChartMoE signifi-
cantly improves the baseline (InternlmXC-v2) performance (72.00% vs. 84.64%, +12.64%↑ in

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

Table 2: The relaxed accuracy (%) performance on ChartQA. Ada.: Adaptive input resolution. *: Multi-scale
image feature, 448×448 in DocOwl. †: Employing token merging to reduce computational overhead.

Models

Para.

Resolution

Relax Acc. @0.05

Relax Acc. @0.10

Relax Acc. @0.20

Human Aug. Avg. Human Aug. Avg. Human Aug. Avg.

LLaVA-v1.5
Qwen-VL
DocOwl-v1.5
InternlmXC-v2

13B
9.6B
8B
8B

Pix2Struct
Matcha
UniChart
Deplot + LLaVA-v1.6

282M
282M
201M
282M+13B

@336
@448
@448*
@490

Ada.
Ada.
@960
Ada.

ChartVLM
OneChart
ChartLlama
ChartGemma+PoT
TinyChart
ChartAst
TinyChart+PoT
ChartMoE (Ours)
ChartMoE+PoT (Ours)

13B

Ada.

125M+13B @1024
@336
@448
@768†
@448
@768†
@490
@490

13B
3B
3B
13B
3B
8B
8B

General MLLMs

25.36
40.48
47.44
62.72

18.56
79.76
91.52
81.28

21.96
60.12
69.48
72.00

Specialist Chart Models
53.48
61.88
58.96
70.56

30.08
37.12
34.64
53.44

76.88
86.64
83.28
87.68

Chart MLLMs

42.08
54.48
58.40
67.84
58.72
64.88
70.24
71.36
78.32

82.48
87.12
93.12
85.28
94.88
93.12
90.72
91.04
90.96

62.28
70.80
75.76
76.56
76.80
79.00
80.48
81.20
84.64

28.56
43.20
51.92
66.72

31.68
39.84
36.48
56.80

43.84
57.60
61.20
68.64
62.56
66.24
71.20
75.12
80.16

23.52
82.56
92.08
84.08

78.40
87.36
84.24
88.48

82.88
87.84
93.60
85.84
95.28
93.84
91.44
92.48
92.32

26.04
62.88
72.00
75.40

55.04
63.60
60.36
72.64

63.36
72.72
77.40
77.24
78.92
80.04
81.32
83.80
86.24

32.56
47.52
56.72
70.80

37.28
43.52
38.88
60.64

46.00
62.00
63.52
69.84
67.04
67.44
72.40
78.16
82.08

30.72
85.76
93.12
86.56

81.12
88.56
85.28
90.08

31.64
66.64
74.92
78.68

59.20
66.04
62.08
75.36

64.64
83.28
75.32
88.64
78.76
94.00
86.32
78.08
96.16 81.60
80.88
94.32
82.48
92.56
93.68 85.92
93.60 87.84

Table 3: The zero-shot performance on ChartBench. No methods are fine-tuned on the trainset for fairness.
We exclude PoT because ChartBench mainly assesses numerical extraction accuracy without math calculation.

Models

LLaVA-v1.5
Qwen-VL
DocOwl-v1.5
Mini-Gemini
InternlmXC-v2

Pix2Struct
Matcha
UniChart
Deplot+LLaVA-v1.6

ChartVLM
ChartLlama
TinyChart
OneChart
ChartGemma
ChartMoE (Ours)

Regular Type

Extra Type

Line

Bar

Pie

Avg. Area

Box Radar Scatter Node Comb. Avg.

29.12
38.00
49.60
34.88
68.16

2.56
6.80
7.04
31.20

21.92
26.80
32.40
41.28
50.48
71.44

21.26
20.71
31.69
36.12
48.74

2.37
5.05
5.35
26.46

14.16
18.83
25.81
30.28
38.21
51.57

22.10
17.28
29.46
38.24
35.68
31.54
40.40
36.77
56.60 54.50

General MLLMs
21.73
28.83
12.27
31.20
27.47

20.94
24.17
23.33
23.33
25.33

27.50
35.00
22.50
30.60
40.10

Specialist Chart Models
0.13
2.33
0.27
5.18
3.86
5.55
21.34
27.09

0.13
1.60
4.80
13.34

4.60
6.20
11.60
24.00

1.90
3.60
4.30
24.00

15.16
10.50
20.99
20.80
26.71
22.50
32.65
29.60
32.10
39.89
52.80 56.31

Chart MLLMs
7.47
14.27
10.13
19.07
28.27
38.40

7.87
12.00
14.80
13.20
24.13
24.13

8.00
24.30
13.40
24.60
28.10
40.20

23.47
19.50
36.13
35.20
52.93

0.67
3.46
5.06
41.34

7.87
27.73
28.14
38.53
48.00
62.67

36.80
18.50
29.60
43.60
50.40

0.40
5.40
15.80
42.00

5.40
26.20
10.80
34.80
41.80
58.00

24.30
25.50
38.80
27.90
46.20

3.20
4.80
9.60
31.00

10.50
25.80
21.60
27.90
43.40
49.20

24.96
26.56
27.38
30.61
39.72

2.93
5.81
8.30
31.57

8.38
21.71
22.56
31.91
42.47
55.58

ALL

23.38
28.18
32.05
34.37
48.41

2.16
4.84
6.78
27.62

11.96
21.31
22.51
29.93
38.46
51.67

Acc.@0.05). Compared to previous SOTA (TinyChart+PoT @768 pixel), ChartMoE consistently
surpasses it across all metrics. The PoT effectively enhances the mathematical reasoning capa-
bilities, which is a common shortfall in current MLLMs. ChartMoE integrates better with PoT,
indicating that it accurately extracts fundamental elements from charts. ChartMoE shows more sig-
nificant improvement on Human part, especially after incorporating PoT, where the questions are
more computationally complex and challenging. Notably, our error analysis in the Augmented part
reveals that many errors stem from limitations of the evaluation criteria, i.e., string matching. For
instance, it is marked incorrect if the prediction is It is between 2003 and 2005 and the ground truth
is (2003, 2005). Forcing performance improvement may lead to model overfitting.

ChartBench. Tab. 3 presents detailed comparisons of ChartMoE on ChartBench. None of the
models, including our ChartMoE, undergo supervised fine-tuning on the ChartBench trainset to
ensure fair experimental comparison. Chart-specific models typically underperform due to limited
generalization, which fails to manage the annotated charts effectively (< 10%). Deplot shows a
distinct advantage over these types of models (27.62%) with the assistance of LLaVA-v1.6. The
baseline (InternlmXC-v2) demonstrates strong generalization on ChartBench (48.41%), which may

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

benefit from pre-training instructions designed for unannotated charts. Without additional design,
ChartMoE improves the baseline performance comprehensively (48.41% vs. 51.67%), especially on
extra chart types (39.72% vs. 55.58%, +15.86%↑).

ChartFC & ChartCheck. Tab. 4 compares ChartMoE on the synthetic ChartFC and real-world
ChartCheck. ChartMoE consistently outperforms SOTA (e.g., ChartGemma +4.4%↑ on ChartFC)
and significantly improves the performance compared to InternlmXC-v2 (+6.83%↑ and +8.76%↑ on
ChartCheck T1 and T2, respectively). Note that this is implemented without using training data for
supervised fine-tuning, demonstrating ChartMoE’s strong generalization capabilities.

Models

ChartFC

PaliGemma
LLaVA-v1.5†
InternlmXC-v2
ChartInstruct-LLama2
ChartInstruct-FlanT5XL
ChartGemma
ChartMoE (Ours)

58.26
61.28
65.93
69.57
70.27
70.33
74.73

ChartCheck

T1

T2

67.34
70.22
72.04
70.11
72.03
71.50
78.87

68.50
70.03
70.44
68.80
73.80
74.31
79.20

Table 4: The accuracy performance
on ChartFC and ChartCheck. †: tun-
ing with ChartGemma instructions.

5 FURTHER STUDY

Figure 4: Training loss of dif-
ferent initialization.

Figure 5: Top-2 selected expert distri-
bution on ChartBench.

5.1 MODEL ARCHITECTURE ABLATION

We investigate the impact of three factors on our MoE connector: the number of experts, the number
of activated experts, and the expert initialization manner. All the experiments are conducted with
ChartQA training data and evaluated on ChartQA test split with relax accuracy metric.

Effect of the Expert Initialization Manner. The initialization strategy plays a crucial role in de-
termining the performance of the MoE connector. Effective initialization is essential to ensure that
each expert performs its designated function optimally. As illustrated in Tab. 5 row 1-3, we ex-
plore the impact of 3 initialization strategies for the MoE connector. Random initialization serves
as a baseline but struggles with convergence (refer to Fig. 4), resulting in a suboptimal accuracy of
73.20% at Acc.@0.05. Following CuMo Li et al. (2024), we employ the Co-Upcycle strategy by
replicating the table-JSON-code aligned connector for all experts. Given the same starting point, this
approach lacks expert diversity, which limits its effectiveness, resulting in an accuracy of 77.48%
at Acc.@0.05. In contrast, our initialization assigns distinct parameters to each expert. This tai-
lored approach enables each expert to capitalize on its specific strengths, resulting in the highest
performance, achieving 78.76% in Acc.@0.05.

Effect of Number of Experts and Activated Experts. As shown in rows 3-4 of Tab. 5, we com-
pare ChartMoE configurations with 4 and 8 experts, keeping 2 experts activated. The 8 experts
are initialized in pairs using the 4 methods illustrated in Fig. 2c. ChartMoE achieves 78.76% in
Acc.@0.05 with 4 experts, which is slightly higher than the 78.60% achieved with 8 experts, show-
ing a marginal increase of +0.16%. In rows 4-5, we compare the performance of configurations with
2 and 4 activated experts, finding similar results: 78.60% vs. 78.64% in Acc.@0.05. This analysis
suggests that merely increasing the number of experts or the activation of experts does not guarantee
improved performance. The configuration with 4 experts and 2 activated experts effectively balances
complexity and performance, making it a suitable choice for ChartMoE.

5.2 TRAINING STRATEGY ABLATION

We analyze the impact of the training strategy across alignment and supervised fine-tuning stages.
We use InternlmXC-v2 with ChartQA fine-tuning as our baseline, maintaining the same hyperpa-
rameters as the chart-specific annealing tuning stage.

Effect of Alignment Strategy. As shown in rows 1-3 of Tab. 6, translating the chart image
into structural text formats such as table, JSON, and code during the alignment stage signifi-
cantly enhances performance in downstream chart understanding tasks. After applying table-JSON-
code alignment, the model achieves 77.20% in Acc.@0.05, representing a notable improvement

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

02004006008001000Iteration02468LossRandomCo-UpcycleChartMoE0%20%40%60%80%100%PercentageAreaBarBoxComb.LineNodePieRadarScatterIXC-v2TableJSONCodeUnder review as a conference paper at ICLR 2025

Table 5: Ablation study on ChartMoE architecture w.r.t.
the total / activated / initialization of connector experts.
All experiments are conducted on ChartQA.

Total
Experts

Activated
Experts

Experts
Initialization

4

4

4

4

8

8

2

2

2

2

2

4

Random Init.

Random Align

Co-Upcycle Init.

Diversely Align

Diversely Align

Diversely Align

Relax Acc @0.05

Human

59.68

62.32

64.96

67.92

67.20

67.68

Aug.

86.72

88.88

90.00

89.60

90.00

89.60

Avg.

73.20

75.60

77.48

78.76

78.60

78.64

Table 6: Ablation study on the proposed training
strategy and connector architecture on the align-
ment, high-quality knowledge learning, and chart-
specific anneal tuning stages.

ChartMoE Recipe

Relax Acc @0.05

Human Aug. Avg.

Baseline: InternlmXC-v2 + ChartQA

63.68

87.68

75.68

+ table-JSON-code Aligned Connector

64.24

90.16

77.20

+ Top2-in-4 ChartMoE Connector

+ MMC High-Quality Knowledge Learning

+ ChartGemma Instructions

67.92

89.60

78.76

67.84

71.36

90.24

79.04

91.04 81.20

Table 7: Ablation study on the expert of
MoE connector. We ignore the gating net-
work and adopt specific expert output.

Table 8: Ablation study on alignment pre-training tasks. We adopt
different alignment tasks for baseline (linear connector) and fur-
ther conduct supervised fine-tuning on the ChartQA train set.

Connector

Expert 0 (Vanilla)
Expert 1 (Table)
Expert 2 (JSON)
Expert 3 (Code)

Relax Acc @0.05

Human

69.76
63.60
60.64
66.88

Aug.

89.84
89.12
82.48
89.36

Avg.

79.80
76.36
71.56
78.12

Alignment

Vanilla
Table
JSON
Code

w/o ChartQA SFT

w/i ChartQA SFT

Human

62.72
51.28
44.40
50.16

Aug.

81.28
71.76
65.12
71.20

Avg.

72.00
61.52
54.76
60.68

Human

63.68
63.92
64.88
64.24

Aug.

87.68
89.28
89.84
90.16

Avg.

75.68
76.60
77.36
77.20

Figure 6: Visualizations of top-1 expert selection. Only the boundaries of the merged tokens are plotted.

(+1.52%↑). When combined with our proposed MoE connector, performance further increases to
78.76%, a total gain of +3.08%↑ in Acc.@0.05.

Effect of Supervise Fine-Tuning Strategy. As shown in rows 4-5 of Tab. 6, we divide the super-
vised fine-tuning stage into two phases. By incorporating high-quality knowledge learning using
the MMC dataset, ChartMoE achieves 79.04% Acc.@0.05, reflecting a 3.36% improvement. In
the chart-specific annealing tuning phase, we introduce ChartGemma data to enhance the model’s
reasoning and PoT capabilities, leading the model to peak performance (81.20%, +5.52%↑).

5.3

IN-DEPTH ANALYSIS

Effect of the Each Expert. To explore the role of each expert in ChartMoE, we bypass the gating
network and manually select the output of specific experts. As shown in Tab. 7, E0 performs the
best (79.80%), which is consistent with the distribution in Fig. 5. However, this doesn’t mean other
experts lack relevance, which may offer crucial insights at key moments (Fig. 6).

Effect of Alignment Task. As shown in Tab. 8, we explore various alignment tasks based on the
linear connector. After alignment, the performance on ChartQA declines compared to the baseline.
However, the aligned model exhibits a substantial improvement after supervised fine-tuning on the
ChartQA train split, which is consistent with previous observations Meng et al. (2024); Yan et al.
(2024). Specifically, the JSON and code tasks exhibit remarkable improvement over the table.

Expert Distribution Visualization. As shown in Fig. 5& 6, we visualize the expert distribution
in the MoE connector on the ChartBench test set. We designate the vanilla connector as E0, while
E1-3 corresponds to connectors aligned with tables, JSON, and code. As depicted in Fig. 5, the
trend is consistent across different chart types, with E0 and E3 being the most frequently selected

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

 Expert initialized with InternlmXC-v2Expert initialized with chart to JSONExpert initialized with chart to tableExpert initialized with chart to codeUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 7: The performance on the general VQA tasks (MME Fu et al. (2023)). With supervised fine-tuning
on extensive chart-structured data, the directly tuned IXC-v2 shows a significant performance drop, while
ChartMoE maintains a satisfying performance by keeping the vanilla connector as the expert in MoE.

Figure 8: The performance on the general VQA tasks
(MMBench Liu et al. (2023e)). Please refer to its paper
for each task’s details. The observations and conclu-
sions are consistent with the MME benchmark.

Figure 9: The performance with/without bz-loss on
ChartQA. Left: The bz-loss leads to more even ex-
pert selections. Right: A more balanced distribution
does not yield better performance.

connectors. The expert selection shows no extreme bias, as even the least chosen, E1, accounts for
over 10%. We further visualize the expert selection for each image token, revealing the preferences
of each expert. As shown in Fig. 6, E0 is the primary choice for background tokens, explaining its
dominance in Fig. 5. E1 and E2 are more frequently chosen by tokens from titles, axis labels, or
legends, as these elements are commonly found in tables and JSON files. ChartMoE tends to use E3
to focus on the data points and visual elements within the chart, e.g., data points on the line, digital
text, and edges in a node chart. These components are essential for accurately re-drawing the charts.

Performance on General Tasks. While ChartMoE is designed to enhance chart understanding,
it does not compromise other capabilities, e.g., instruction following and object recognition.
In
Fig. 7& 8, we show the comparisons of directly fine-tuned InternlmXC-v2 (short for Direct SFT)
with data from Tab. 1 and the baseline (short for IXC-v2) on general benchmarks Fu et al. (2023); Liu
et al. (2023e). The direct SFT model shows diminished general capabilities. In contrast, ChartMoE
preserves it nearly intact by retaining the original connector as one of its experts.

Effect of Balance Loss in MoE. The standard MoE Zoph et al. (2022) employs balanced loss and
router z-loss (short for bz-loss) to prevent certain experts from dominating the model training. In
Fig. 9, we compare the effects of with and without bz-loss. While bz-loss promotes a more equitable
selection of experts, it fails to enhance ChartMoE’s performance further. As shown in Fig. 6, the
expert initialization in ChartMoE results in each expert having its own preference for visual token
selection (refer to Appendix B for detail). Consequently, the bz-loss might hinder the model’s
convergence to the optimal point because the distribution of visual tokens is inherently imbalanced.

6 CONCLUSION

We introduce ChartMoE, a multi-task aligned and instruction-tuned MLLM designed for complex
chart understanding and reasoning. We replace the linear connector with the MoE architecture and
initialize each expert with parameters derived from different alignment tasks. We further present the
ChartMoE-Align dataset, a synthetic collection of nearly 1 million table-json-code-chart quadruples,
to facilitate alignment training across different experts. This approach preserves the strengths of each
alignment task, ensuring efficient training and superior model performance. ChartMoE outperforms
the previous state-of-the-art on several benchmarks by a large margin and excels in real-world ap-
plications such as chart question answering, translation, and editing. Please refer to Appendix A.3
for the reproducibility statement.

10

ColorExistenceLandmarkScenePositionOCRArtworkPostersCelebrityCountCommonsenseCodeCalculationTranslationPerceptionCognitionTotal80100120140160180200220Fine-grained ScoresDirect SFTChartMoEIXC-v2050010001500200025003000Total ScoresDirect SFTChartMoEIXC-v2A_OverallB_ARB_CPB_FP-CB_FP-SB_LRB_RR5060708090100Fine-grained ScoresDirect SFTChartMoEIXC-v2RandomCo-Upcyclew/i bz-lossw/o bz-lossRandomCo-Upcyclew/i bz-lossw/o bz-loss020406080100Percentage (%)Expert 0Expert 1Expert 2Expert 3727374757677787980Average accuracy on ChartQA73.277.4878.2878.76Under review as a conference paper at ICLR 2025

REFERENCES

Mubashara Akhtar, Oana Cocarascu, and Elena Paslaru Bontas Simperl. Reading and reasoning over chart

images for evidence-based automated fact-checking. arXiv preprint:2301.11843, 2023a.

Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, and Elena Sim-
arXiv

Chartcheck: An evidence-based fact-checking dataset over real-world chart

images.

perl.
preprint:2311.07453, 2023b.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: a visual language model for few-shot

learning. In proceedings of NeurIPS, volume 35, pp. 23716–23736, 2022.

Alibaba. Qwen-vl 2.5: A multimodal large language model. https://tongyi.aliyun.com/qianwen,

2024. Accessed: 2024-09-17.

Jinze Bai, Shuai Bai, Yunfei Chu, et al. Qwen technical report. arXiv preprint:2309.16609, 2023a.

Jinze Bai, Shuai Bai, Shusheng Yang, et al. Qwen-vl: A frontier large vision-language model with versatile

abilities. arXiv preprint:2308.12966, 2023b.

Lucas Beyer, Andreas Steiner, Andr´e Susano Pinto, et al. Paligemma: A versatile 3b vlm for transfer. arXiv

preprint:2407.07726, 2024.

Tom Brown, Benjamin Mann, Nick Ryder, et al. Language models are few-shot learners. In proceedings of

NeurIPS, volume 33, pp. 1877–1901, 2020.

Victor Carbune, Hassan Mansoor, Fangyu Liu, et al. Chart-based reasoning: Transferring capabilities from

llms to vlms. In proceedings of NAACL, 2024.

Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui
Han, and Xiangyu Zhang. Onechart: Purify the chart structural extraction via one auxiliary token. arXiv
preprint:2404.09987, 2024.

Jun Chen, Deyao Zhu, Xiaoqian Shen, et al. Minigpt-v2:

large language model as a unified interface for

vision-language multi-task learning. arXiv preprint:2310.09478, 2023a.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentan-

gling computation from reasoning for numerical reasoning tasks. TMLR, 2023b.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, et al. Internlm-xcomposer2: Mastering free-form text-image compo-

sition and comprehension in vision-language large model. arXiv preprint:2401.16420, 2024.

Chaoyou Fu, Peixian Chen, Yunhang Shen, et al. Mme: A comprehensive evaluation benchmark for multimodal

large language models. arXiv preprint:2306.13394, 2023.

Yucheng Han, Chi Zhang, Xin Chen, et al. Chartllama: A multimodal llm for chart understanding and genera-

tion. arXiv preprint:2311.16483, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In

proceedings of CVPR, pp. 770–778, 2016.

Enamul Hoque, Vidya Setlur, Melanie Tory, and Isaac Dykeman. Applying pragmatics principles for interaction

with visual analytics. IEEE TVCG, 24(1):309–318, 2017.

Ting-Yao Hsu, C Lee Giles, and Ting-Hao’Kenneth’ Huang. Scicap: Generating captions for scientific figures.

In Findings of ACL, 2021.

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei
Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv
preprint:2403.12895, 2024.

Edward J Hu, Yelong Shen, Phillip Wallis, et al. LoRA: Low-rank adaptation of large language models. In

proceedings of ICLR, 2022.

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, et al. Sparse upcycling: Training mixture-of-experts

from dense checkpoints. In proceedings of ICLR, 2023.

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, et al. Pix2struct: Screenshot parsing as pretraining for visual

language understanding. In proceedings of ICML, pp. 18893–18912, 2023.

11

540
541
542
543

544
545
546
547
548
549

550
551
552
553
554

555
556
557
558
559
560

561
562
563
564
565
566

567
568
569
570
571
572

573
574
575
576
577

578
579
580
581
582
583

584
585
586
587
588
589

590
591
592
593

Under review as a conference paper at ICLR 2025

594
595
596
597

598
599
600
601
602
603

604
605
606
607
608

609
610
611
612
613
614

615
616
617
618
619
620

621
622
623
624
625
626

627
628
629
630
631

632
633
634
635
636
637

638
639
640
641
642
643

644
645
646
647

Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin
Wen. Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts. arXiv preprint:2405.05949,
2024.

Junnan Li, Dongxu Li, Silvio Savarese, et al. BLIP-2: bootstrapping language-image pre-training with frozen
image encoders and large language models. In proceedings of ICML, volume 202, pp. 19730–19742, 2023.

Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan.
Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024.

Ziyi Lin, Chris Liu, Renrui Zhang, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings

for multi-modal large language models. arXiv preprint:2311.07575, 2023.

Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, et al. Deplot: One-shot visual

language reasoning by plot-to-table translation. In Findings of ACL, pp. 10381–10399, 2023a.

Fangyu Liu, Francesco Piccinno, Syrine Krichene, et al. Matcha: Enhancing visual language pretraining with

math reasoning and chart derendering. In proceedings of ACL, pp. 12756–12770, 2023b.

Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and
Dong Yu. MMC: advancing multimodal chart understanding with large-scale instruction tuning. In proceed-
ings of ACL, 2023c.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In proceedings of

NeurIPS, 2023d.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next:
Improved reasoning, ocr, and world knowledge, January 2024a. URL https://llava-vl.github.
io/blog/2024-01-30-llava-next/.

Haotian Liu, Chunyuan Li, Yuheng Li, et al. Improved baselines with visual instruction tuning. In proceedings

of CVPR, 2024b.

Mengsha Liu, Daoyuan Chen, Yaliang Li, Guian Fang, and Ying Shen. Chartthinker: A contextual chain-of-

thought approach to optimized chart summarization. arXiv preprint:2403.11236, 2024c.

Yuan Liu, Haodong Duan, Yuanhan Zhang, et al. Mmbench: Is your multi-modal model an all-around player?

arXiv preprint:2307.06281, 2023e.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren,
towards real-world vision-language understanding. arXiv

Zhuoshu Li, Hao Yang, et al. Deepseek-vl:
preprint:2403.05525, 2024.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, et al. Chartqa: A benchmark for question answering about charts

with visual and logical reasoning. In proceedings of ACL, 2022.

Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. Unichart: A universal
vision-language pretrained model for chart comprehension and reasoning. In proceedings of EMNLP, 2023.

Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartinstruct:

Instruction tuning for chart comprehension and reasoning. arXiv preprint:2403.09028, 2024a.

Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chartgemma:

Visual instruction-tuning for chart reasoning in the wild. arXiv preprint:2407.04172, 2024b.

Fanqing Meng, Wenqi Shao, Quanfeng Lu, et al. Chartassisstant: A universal chart multimodal language model

via chart-to-table pre-training and multitask instruction tuning. arXiv preprint:2401.02384, 2024.

Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific

plots. In proceedings of CVPR, pp. 1527–1536, 2020.

OpenAI. Gpt-4 technical report. arXiv preprint:2303.08774, 2023.

OpenAI. Gpt-4o: A multimodal large language model. https://openai.com, 2024. Accessed: 2024-09-

17.

Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, et al. Dinov2: Learning robust visual features without

supervision. TMLR, 2023.

12

Under review as a conference paper at ICLR 2025

648
649
650
651

652
653
654
655
656
657

658
659
660
661
662

663
664
665
666
667
668

669
670
671
672
673
674

675
676
677
678
679
680

681
682
683
684
685

686
687
688
689
690
691

692
693
694
695
696
697

698
699
700
701

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by

generative pre-training. OpenAI blog, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language

supervision. In proceedings of ICML, pp. 8748–8763. PMLR, 2021.

Gemma Team, Thomas Mesnard, Cassidy Hardin, et al. Gemma: Open models based on gemini research and

technology. arXiv preprint:2403.08295, 2024a.

Gemma Team, Thomas Mesnard, Cassidy Hardin, et al. Gemma: Open models based on gemini research and

technology. arXiv preprint:2407.21783, 2024b.

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan
Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration
of multimodal llms. arXiv preprint arXiv:2406.16860, 2024a.

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring

the visual shortcomings of multimodal llms. In Proceedings of CVPR, pp. 9568–9578, 2024b.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models.

arXiv preprint:2302.13971, 2023.

Peifang Wang, Olga Golovneva, Armen Aghajanyan, et al. DOMINO: A dual-system for multi-step visual

language reasoning. arXiv preprint:2310.02804, 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large

language models. In proceedings of NeurIPS, volume 35, pp. 24824–24837, 2022.

Yifan Wu, Lutao Yan, Yuyu Luo, Yunhai Wang, and Nan Tang. Evaluating task-based effectiveness of mllms

on charts. arXiv preprint:2405.07001, 2024.

Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, et al. Chartx & chartvlm: A versatile benchmark and

foundation model for complicated chart reasoning. arXiv preprint:2402.12185, 2024.

Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for

complex visual reasoning in charts. arXiv preprint:2312.15915, 2023.

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu,
Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv
preprint:2408.08872, 2024.

Pengyu Yan, Mahesh Bhosale, Jay Lal, Bikhyat Adhikari, and David S. Doermann. Chartreformer: Natural

language-driven chart image editing. arXiv preprint:2403.00209, 2024.

Jiabo Ye, Anwen Hu, Haiyang Xu, et al. Ureader: Universal ocr-free visually-situated language understanding

with multimodal large language model. In Findings of ACL, 2023a.

Qinghao Ye, Haiyang Xu, Guohai Xu, et al. mplug-owl: Modularization empowers large language models with

multimodality. arXiv preprint:2304.14178, 2023b.

Qinghao Ye, Haiyang Xu, Jiabo Ye, et al. mplug-owl2: Revolutionizing multi-modal large language model

with modality collaboration. arXiv preprint:2311.04257, 2023c.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-

training. In proceedings of ICCV, pp. 11975–11986, 2023.

Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tiny-
chart: Efficient chart understanding with visual token merging and program-of-thoughts learning. arXiv
preprint:2404.16635, 2024.

Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, et al. Internlm-xcomposer: A vision-language
large model for advanced text-image comprehension and composition. arXiv preprint:2309.15112, 2023.

Susan Zhang, Stephen Roller, Naman Goyal, et al. Opt: Open pre-trained transformer language models. arXiv

preprint:2205.01068, 2022.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.

In proceedings of NeurIPS, 2023.

Li Zhuowan, Jasani Bhavan, Tang Peng, and Ghadar Shabnam. Synthesize step-by-step: Tools, templates and

llms as data generators for reasoning-based chart vqa. In proceedings of CVPR, 2024.

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam M. Shazeer, and William
Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint:2202.08906, 2022.

13

Under review as a conference paper at ICLR 2025

A ADDITIONAL EXPERIMENTAL SETTINGS AND RESULTS

A.1 TOP-2 EXPERTS DISTRIBUTION

Our ChartMoE employs MoE connector expert parameters initialized with various alignment tasks. To in-
vestigate the impact of these initialization methods on model performance, we present the comparisons in
Tab. 6& 7& 8 and Fig. 4&5. For a deeper analysis, we explore how different initialization methods affect
expert selection. As shown in Fig. 10, both random initialization and co-upcycle result in a more uniform
distribution of experts. However, this uniformity does not inherently lead to improved performance or inter-
pretability, possibly due to insufficient differentiation among the experts. In contrast, our ChartMoE clearly
prefers specialized roles, as illustrated in Fig. 6& 11& 12.

(a) Random

(b) Co-Upcycle

(c) ChartMoE with bz-loss

(d) ChartMoE

Figure 10: The distribution of Top-2 experts after supervised fine-tuning with three expert initialization meth-
ods. We calculate the proportion of the top 2 experts selected by the router on the ChartBench.

A.2 SUMMARY OF HYPERPARAMETER SETTINGS

The training process of our ChartMoE is structured into three distinct phases: Alignment Pre-training, High-
Quality Knowledge Learning, and Chart-Specific Annealing Tuning. Table 9 provides a comprehensive
overview of the hyperparameter configurations employed during each training stage.

Table 9: Training hyperparameters of ChartMoE for all stages.

Configuration

Alignment Pre-training

High-Quality Knowledge Learning

Chart Specific Annealing Tuning

Connector Initialization

InternlmXC-v2

Table&JSON&Code Experts + InternlmXC-v2

ChartMoE 2nd-stage

LLM Training

Image Resolution

ViT Sequence Length

Freeze

490

1225

LoRA

490

1225

LoRA

490

1225

Optimizer
Optimizer Hyperparameter β1 = 0.9, β2 = 0.95, ϵ = 1e−8
Peak Learning Rate

AdamW

5e−5

AdamW
β1 = 0.9, β2 = 0.95, ϵ = 1e−8
1e−5

AdamW
β1 = 0.9, β2 = 0.95, ϵ = 1e−8
5e−5

Learning Rate Schedule

cosine decay

cosine decay

cosine decay

Weight Decay

Gradient Clip

Warm-up Ratio

Global Batch Size

Gradient Acc.

Numerical Precision

Optimizer Sharding

Gradient Sharding

Parameter Sharding

Activation Checkpointing

GPU Hours (A100-40G)

0.1

1.0

0.01

64

8

bfloat16
✓

✓

×
✓

240

0.1

1.0

0.01

32

8

bfloat16
✓

✓

×
✓

138

0.1

1.0

0.01

32

8

bfloat16
✓

✓

×
✓

76

A.3 REPRODUCIBILITY STATEMENT

We have included the architecture of ChartMoE in Section 3.1 and the complete training procedure in Section
3.2 and Section 3.3. The training data recipe is listed in Tab. 1 in detail. Hyper-parameter settings are shown
in Appendix A.2. We also introduce the generation pipeline for ChartMoE-Align in Section 3.2, and some
detailed examples in Appendix C. Furthermore, our ChartMoE-Align dataset and checkpoints of ChartMoE
will be released soon on GitHub and Huggingface.

14

702
703
704
705

706
707
708
709
710
711

712
713
714
715
716

717
718
719
720
721
722

723
724
725
726
727
728

729
730
731
732
733
734

735
736
737
738
739

740
741
742
743
744
745

746
747
748
749
750
751

752
753
754
755

0%20%40%60%80%100%PercentageAreaBarBoxComb.LineNodePieRadarScatterExpert 0Expert 1Expert 2Expert 30%20%40%60%80%100%PercentageAreaBarBoxComb.LineNodePieRadarScatterExpert 0Expert 1Expert 2Expert 30%20%40%60%80%100%PercentageAreaBarBoxComb.LineNodePieRadarScatterIXC-v2TableJSONCode0%20%40%60%80%100%PercentageAreaBarBoxComb.LineNodePieRadarScatterIXC-v2TableJSONCodeUnder review as a conference paper at ICLR 2025

B ADDITIONAL VSUALIZATIONS OF TOP-1 EXPERT SELECTION

In this section, we randomly sampled images from natural image datasets (LLaVA-CC3M Liu et al. (2023d))
and chart datasets (ChartQA Masry et al. (2022), ChartBench Xu et al. (2023)) to illustrate ChartMoE’s token
selection preferences. As shown in Fig. 11, the vanilla expert focuses more on the background, the table expert
concentrates on details such as the boundary between the background and the subject, the JSON expert focuses
on textures (e.g., maps and objects), and the code expert specializes in curves and trends (e.g., logos and text).
Fig. 12 further demonstrates that while the vanilla expert continues to attend to background tokens, critical
visual elements are handled by the aligned experts, with the code expert being notably more prominent.

Figure 11: More visualizations of top-1 expert selection on general images randomly sampled from LLaVA-
CC3M. These examples show the selection preferences of different experts in ChartMoE.

15

756
757
758
759

760
761
762
763
764
765

766
767
768
769
770

771
772
773
774
775
776

777
778
779
780
781
782

783
784
785
786
787
788

789
790
791
792
793

794
795
796
797
798
799

800
801
802
803
804
805

806
807
808
809

CSV TableFilter out meta CSV data from existing datasets Generate meta CSV data by larege language modelsAttribution JSONRandom sample from predefined attribution setsInversion from chart by finetuned Deplot modelChartExecute renderGet executable code with predefined templates and json propertiesPython CodeFailedSuccessRenderKeep the quadrupleDiscard the quadruple Expert initialized with InternlmXC-v2Expert initialized with chart to tableExpert initialized with chart to JSONExpert initialized with chart to codeUnder review as a conference paper at ICLR 2025

810
811
812
813

814
815
816
817
818
819

820
821
822
823
824

825
826
827
828
829
830

831
832
833
834
835
836

837
838
839
840
841
842

843
844
845
846
847

848
849
850
851
852
853

854
855
856
857
858
859

860
861
862
863

Figure 12: More visualizations of top-1 expert selection on chart images. The vanilla expert primarily handles
background tokens, and the chart visual markers are handled by other experts.

16

CSV TableFilter out meta CSV data from existing datasets Generate meta CSV data by larege language modelsAttribution JSONRandom sample from predefined attribution setsInversion from chart by finetuned Deplot modelChartExecute renderGet executable code with predefined templates and json propertiesPython CodeFailedSuccessRenderKeep the quadrupleDiscard the quadruple Expert initialized with InternlmXC-v2Expert initialized with chart to tableExpert initialized with chart to JSONExpert initialized with chart to codeUnder review as a conference paper at ICLR 2025

C DETAILS OF CHARTMOE-ALIGN

C.1 OVERVIEW

ChartMoE-Align is a dataset we introduced for different experts aligning pretraining. It consists of nearly 1
million Chart Table JSON Code quadruples and supports three alignment tasks: Chart to Table, Chart to JSON,
and Chart to Code. Unlike other chart datasets, ChartMoE-Align focuses solely on these three fundamental
alignment tasks without considering the diversity of instruction tasks.

C.2 TABLE DATA COLLECTION

We primarily collect table data from three sources: the ChartQA training set Masry et al. (2022), the PlotQA
training set Methani et al. (2020), and ChartY provided by OneChart Chen et al. (2024).

ChartQA includes 18.3K real-world charts and provides accompanying meta tables. While the charts are of
high quality and manually curated, they lack fine-grained attribute annotations and executable plotting code.
As a result, we only retained the tables from ChartQA in CSV format.

PlotQA comprises 157K charts, primarily focusing on three common types: line, bar, and pie charts. These
charts are generated using Python code with limited formatting and style diversity. Consequently, we did not
utilize the charts from PlotQA but retained its 157K tables. These tables originate from sources like World
Bank Open Data, Open Government Data, and the Global Terrorism Database, covering statistics on various
indicators such as fertility rates, rainfall, coal production, and more across years, countries, and districts.

ChartY is a chart dataset containing 2.7M charts in both Chinese and English proposed by OneChart. Notably,
ChartY also includes charts from ChartQA and PlotQA, which we filtered out in ChartMoE-Align. Addition-
ally, ChartY primarily consists of common chart types such as line, bar, and pie charts (or their combinations)
and suffers from significant data imbalance. To address this, we sampled a subset to ensure a roughly equal
number of charts for each type. As the tables in ChartY are mainly generated by GPT-3.5 based on templates,
we ultimately retained 763K samples from this source.

C.3 PAIR DATA CONSTRUCTION

JSON provides a structured format distinct from CSV, designed to retain chart attributes beyond numerical
data, such as title position, font size, element colors, legend styles, and more. We adopt the template provided
by ChartReformer Yan et al. (2024) and further enhance it. We add chart type-agnostic attributes like title
position and gridlines. For chart type-specific attributes, we aim to remain consistent with ChartReformer’s
definitions while accommodating all chart types present in ChartMoE-Align. With this framework, we generate
corresponding JSON files for all tables. To extract chart type-specific attributes, we fine-tune a Deplot Liu et al.
(2023a) model, leveraging the original chart to extract their properties. Missing attributes are filled in using
random sampling to ensure completeness.

Code refers to Python scripts based on matplotlib for rendering the charts. Leveraging the rich attributes defined
in the JSON, the code is designed to faithfully represent every attribute to ensure diversity in the resulting charts.
During generation, we explicitly specify all default parameters, such as the hexadecimal color codes for each
line/bar, default font sizes, text positions, etc. We provide basic code templates for type-agnostic attributes. For
type-specific attributes, rules are used to automatically generate the corresponding code.

Chart is produced by executing the generated code. Given the number of table, JSON, and code pairs, we filter
out any quadruples with execution errors or warnings during the chart generation process, retaining only valid
and error-free samples.

Instruction. Considering the alignment task, we directly employ several templated questions to define the
Chart-to-X tasks (X is the ground truth). Ultimately, each quadruple corresponds to three QA pairs. Note that
ChartMoE-Align only serves for alignment training to initialize different expert projectors, thus emphasizing
the diversity of charts and aligned modalities. To improve model performance and instruction-following, we
still require more diverse instructions for supervised fine-tuning to update the MoE connector and LLM.

C.4 QUALITY CONTROL

We first remove all duplicate entries from the meta table and then eliminate quadruples that cause errors or
warnings during rendering. To further assess the quality of ChartMoE-Align, we randomly sample 200 quadru-
ples and ask GPT-4o and annotation experts (with at least three experts reviewing each quadruple) to evaluate
the clarity and readability of the charts, as well as the alignment between the charts and table/JSON/code,
scoring them as 0 or 1. The results show that nearly all charts are clear, unambiguous, and free from obstruc-
tions (GPT-4o: 96.5%, Experts: 99%). Over 90% of the pairs are matching and suitable for instruction tuning
(GPT-4o: 91%, Experts: 94.5%).

17

864
865
866
867

868
869
870
871
872
873

874
875
876
877
878

879
880
881
882
883
884

885
886
887
888
889
890

891
892
893
894
895
896

897
898
899
900
901

902
903
904
905
906
907

908
909
910
911
912
913

914
915
916
917

Under review as a conference paper at ICLR 2025

C.5 EXAMPLE VISUALIZATION

(a) Charts in ChartMoE-Align.

(b) Tables in ChartMoE-Align.

(c) JSONs in ChartMoE-Align. JSON is combined with the table during alignment pre-training.

(d) Codes in ChartMoE-Align. All values and attributes are expressed explicitly.
Figure 13: Detailed Examples in ChartMoE-Align. Each quadruple contains the chart, table, JSON and code.

18

918
919
920
921

922
923
924
925
926
927

928
929
930
931
932

933
934
935
936
937
938

939
940
941
942
943
944

945
946
947
948
949
950

951
952
953
954
955

956
957
958
959
960
961

962
963
964
965
966
967

968
969
970
971

Characteristic,Pharmaceuticals and Vaccines,Consumer Healthcare2020,24038,100332019,24711,89952018,23163,76582017,22436,77502016*,20696,71932015*,17813,60382014*,18597,43222013*,20743,47132012*,20645,47312011,21894,54032010,23284,51082009,23653,47152008,20381,39712007,19163,35532006,20013,3212  Characteristic,Male,Female2019,486,1442018,492,1662017,492,1732016,461,1552015,432,1782014,371,1522013,358,1512012,391,1572011,427,1792010,401,1542009,448,1622008,467,1472007,431,1662006,445,1622005,484,1802004,425,2002003,394,1572002,375,2072001,391,1622000,397,149  Characteristic,"Deaths per 1,000 inhabitants"Somalia,10.86Mauritania,7.22Comoros,7.21Sudan,7.19Djibouti,7.1Tunisia,6.26Yemen,5.98Egypt,5.82Syria,5.37Libya,5.1Morocco,5.06Iraq,4.78Algeria,4.72Lebanon,4.36Jordan,3.86Saudi Arabia,3.47West Bank and Gaza,3.46Kuwait,2.7Oman,2.44Bahrain,2.39United Arab Emirates,1.47Qatar,1.2  Characteristic,Domestic,International2020-2040**,1.9%,4.2%2019-2039**,1.7%,4.5%2018-2038**,1.9%,4.6%2017-2037**,0.9%,3.5%2016-2036**,1.3%,3.8%2020,1.9%,6.2%2019,2.8%,-1.3%2018,7.7%,10%2017,9.5%,9.7%2016,2.1%,-1.3%2015,3.3%,0.8%2014,2.3%,0.3%2013,0.7%,-7.5%2012,2.1%,-3.8%2011,-6.1%,9.4%  {  "type_agnostic": {    "x_font_name": "sans-serif",    "x_font_size": "x-large",    "y_font_name": "monospace",    "y_font_size": "x-large",    "x_tick_size": "small",    "x_tick_rotation": 45,    "y_tick_size": "small",    "legend_loc": "lower left",    "legend_ncols": 3,    "legend_font_size": "medium",    "title_font_name": "monospace",    "title_font_size": "medium",    "grid_vis": false,    "grid_axi": "x",    "grid_which": "minor",    "grid_line_style": "solid",    "vis_tool": "default"  },  "type_specific": {    "colormap": "Blues",    "hatch": "*",    "align": "center"  },  "layout": {    "title": "",    "plot_labels": [    "Pharmaceuticals and Vaccines",    "Consumer Healthcare"    ]  }} {  "type_agnostic": {    "x_font_name": "Serif",    "x_font_size": "large",    "y_font_name": "sans-serif",    "y_font_size": "medium",    "x_tick_size": "x-small",    "x_tick_rotation": 0,    "y_tick_size": "large",    "legend_loc": "lower center",    "legend_ncols": 2,    "legend_font_size": "x-small",    "title_font_name": "monospace",    "title_font_size": "medium",    "grid_vis": true,    "grid_axi": "y",    "grid_which": "minor",    "grid_line_style": "solid",    "vis_tool": "ggplot"  },  "type_specific": {    "colormap": "turbo",    "marker": "s",    "style": "--",    "linewidth": 1.0,    "markersize": 10  },  "layout": {    "title": "",    "plot_labels": [    "Male",    "Female"    ]  }} {  "type_agnostic": {    "x_font_name": "sans-serif",    "x_font_size": "large",    "y_font_name": "Serif",    "y_font_size": "x-large",    "x_tick_size": "small",    "x_tick_rotation": 45,    "y_tick_size": "small",    "legend_loc": "upper center",    "legend_ncols": 3,    "legend_font_size": "x-small",    "title_font_name": "Serif",    "title_font_size": "medium",    "grid_vis": false,    "grid_axi": "both",    "grid_which": "major",    "grid_line_style": "dashed",    "vis_tool": "ggplot"  },  "type_specific": {    "colormap": "plasma",    "hatch": ".",    "align": "center"  },  "layout": {    "title": "",    "plot_labels": []  }} {  "type_agnostic": {    "x_font_name": "sans-serif",    "x_font_size": "large",    "y_font_name": "monospace",    "y_font_size": "medium",    "x_tick_size": "large",    "x_tick_rotation": 45,    "y_tick_size": "medium",    "legend_loc": "lower left",    "legend_ncols": 3,    "legend_font_size": "x-small",    "title_font_name": "Serif",    "title_font_size": "x-large",    "grid_vis": true,    "grid_axi": "x",    "grid_which": "major",    "grid_line_style": "dashed",    "vis_tool": "default"  },  "type_specific": {    "colormap": "cividis",    "hatch": "\\",    "align": "center"  },  "layout": {    "title": "",    "plot_labels": [    "Domestic",    "International"    ]  }} import matplotlib.pyplot as pltimport numpy as np # vis toolplt.style.use('default') # datax = ['2019', '2018', '2017', '2016*', '2015*', '2014*', '2013*', '2012*', '2011', '2010', '2009', '2008', '2007', '2006']y = [[24711, 23163, 22436, 20696, 17813, 18597, 20743, 20645, 21894, 23284, 23653, 20381, 19163, 20013], [8995, 7658, 7750, 7193, 6038, 4322, 4713, 4731, 5403, 5108, 4715, 3971, 3553, 3212]] plt.figure(figsize=(10, 6)) # a vertical bar chartplt.bar(x, y[0], label="Pharmaceuticals and Vaccines", color='#b0d2e7', hatch='*', align='center')plt.bar(x, y[1], bottom=np.sum(y[:1], axis=0), label="Consumer Healthcare", color='#66abd4', hatch='*', align='center') # set the tick of x/yplt.xticks(fontsize='small', rotation=45)plt.yticks(fontsize='small') # set the global legendplt.legend(loc='lower left', ncol=3, fontsize='medium') # set the gridplt.grid(visible=False) # Automatically resize the image by tight_layout()plt.tight_layout()# save the chartplt.savefig('output.png')# Clear the current image stateplt.clf()  import matplotlib.pyplot as pltimport numpy as np # vis toolplt.style.use('ggplot') # datax = [2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018]y = [[394, 425, 484, 445, 431, 467, 448, 401, 427, 391, 358, 371, 432, 461, 492, 492], [157, 200, 180, 162, 166, 147, 162, 154, 179, 157, 151, 152, 178, 155, 173, 166]] plt.figure(figsize=(10, 6)) # a line chartplt.plot(x,y[0], label="Male", color='#466be3', marker='s', markersize=10, linestyle='--', linewidth=1.0)plt.plot(x,y[1], label="Female", color='#30123b', marker='s', markersize=10, linestyle='--', linewidth=1.0) # set the tick of x/yplt.xticks(fontsize='x-small', rotation=0)plt.yticks(fontsize='x-small') # set the global legendplt.legend(loc='lower center', ncol=2, fontsize='x-small') # set the gridplt.grid(visible=True, which='minor', linestyle='solid', axis='y') # Automatically resize the image by tight_layout()plt.tight_layout()# save the chartplt.savefig('output.png')# Clear the current image stateplt.clf() Under review as a conference paper at ICLR 2025

D FURTHER DISCUSSION

D.1 CONTRIBUTION OF CHARTMOE

Some prior work, such as MoE-LLaVA Lin et al. (2024), DeepSeek-VL Lu et al. (2024), and CuMo Li et al.
(2024), has employed MoE architectures in MLLMs. However, these approaches all apply MoE to LLMs or
ViTs to increase model capacity, introducing a large number of learnable parameters to boost performance. In
contrast, our ChartMoE introduces several distinctive innovations:

1) Motivation: Our goal is not to expand model capacity but to enhance the model’s chart comprehension
through alignment tasks while preserving performance on other general tasks. Hence, we retain the original
connector parameters as one expert initialization manner.

2) Initialization: Unlike previous methods that rely on random or co-upcycle initialization, we leverage multi-
ple alignment tasks for expert (connector) initialization. This approach enables ChartMoE to exhibit remarkable
interpretability (Fig. 6& 11& 12).

3) Complexity: We are the first to apply MoE exclusively to the MLP connector (projector) in LLaVA-like
MLLMs. In ChartMoE (based on InternlmXC-v2), the MoE architecture introduces minimal additional param-
eters (model size 8.364B → 8.427B, + 63M↑ only) and training complexity (Fig. 4). It also shows negligible
impact on inference speed (0.945 → 0.952 seconds per QA on ChartQA test set) and peak memory usage (23.72
GB → 23.86 GB, fp16 on A100-40G GPU).

D.2 CHARTMOE BASED ON OTHER MLLMS

Our ChartMoE is based on InterlmXC-v2, but our proposals (MoE connector, diverse alignment, etc.) are
general approaches. Therefore, we use 10% of the alignment data (Tab. 1) and the ChartQA training data to
train our proposals based on LLaVA-v1.5-7B to further demonstrate their effectiveness. As shown in Tab. 10,
our proposals significantly improve the base model. This is partly because LLaVA is trained with fewer chart
data, leading to a lower baseline, and also indicates that the additional alignment data greatly enhances chart
understanding.

Table 10: Performance comparison on ChartQA with LLaVA-v1.5-7B as base MLLM.

Models

LLaVA-v1.5-7B
LLaVA-v1.5-7B + ChartQA
LLaVA-v1.5-7B + ChartMoE

Relax Acc @0.05

Relax Acc @0.10

Relax Acc @0.20

Human

7.60
6.08
18.13

Aug

7.36
23.04
32.11

Avg

7.48
14.56
25.12

Human

7.92
8.24
20.20

Aug

8.08
32.96
42.32

Avg

8.00
20.60
31.36

Human

9.04
10.32
24.24

Aug

9.52
42.16
52.12

Avg

9.28
26.24
38.18

D.3 PERFORMANCE ON CHARTQA

In Tab. 2, our ChartMoE significantly outperforms SOTA. However, some models perform better than ours
on the Augment part of the ChartQA test set. Given that the Augment part of ChartQA is considerably easier
than the Human part, we conduct a more detailed analysis. We analyze the performance of various models on
numeric (Human: 43%, Augment: 39%) and non-numeric (Human: 57%, Augment: 61%) questions. As shown
in Tab. 11, ChartMoE excels in all subcategories except for non-numeric questions in the Augment part. We
find that ChartMoE’s errors primarily occur in string-matching tasks. For instance, a prediction of It is between
2003 and 2005 is marked incorrect if the ground truth is (2003, 2005). High accuracy in this category may
indicate overfitting instead.

Table 11: Fine-grained performance comparison on ChartQA with error margin 5%.

Method

Human

Augment

Numeric

Non-Numeric

Avg

Numeric

Non-Numeric

Avg

TinyChart
ChartAst
ChartMoE (Ours)

58.52%
67.04%
73.89%

58.03%
65.35%
75.49%

58.24%
66.08%
74.80%

92.43%
93.20%
93.20%

96.25%
93.07%
90.98%

94.32%
93.12%
91.84%

Acc

76.28%
79.00%
84.64%

D.4 LIMITATIONS

ChartMoEhas two limitations: 1) Dependency on alignment tasks. ChartMoE requires chart-Table/JSON/Code
alignment tasks for initialization. Non-chart multimodal tasks need new alignment designs to initialize MoE
experts. 2) Limited flexibility. Modifying the projector into a multi-expert architecture makes ChartMoE non-
plug-and-play like LoRA. We are required to retrain the router network when new experts are coming.

19

972
973
974
975

976
977
978
979
980
981

982
983
984
985
986

987
988
989
990
991
992

993
994
995
996
997
998

999
1000
1001
1002
1003
1004

1005
1006
1007
1008
1009

1010
1011
1012
1013
1014
1015

1016
1017
1018
1019
1020
1021

1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

E CAPABILITY COMPARISONS ON CHART EDITING AND HIGHLIGHTING

ChartMoE, leveraging carefully designed alignment training, excels in accurately extracting chart values and
attributes, facilitating precise image reconstruction, chart element editing, and highlighting. In this section, we
provide a visual comparison with the SOTA MLLMs, Qwen-2.5 Alibaba (2024), and GPT-4O OpenAI (2024).
While all three models perform the required instructions effectively, ChartMoE demonstrates superior accuracy
in value and attribute extraction.

Figure 14: Chat demo involves modifying chart types and values. All models successfully convert the chart
type, but only ChartMoE handles the legend correctly. No model makes errors in this task due to the simplicity
of the values and the presence of data point labels.

20

1026
1027
1028
1029

1030
1031
1032
1033
1034
1035

1036
1037
1038
1039
1040

1041
1042
1043
1044
1045
1046

1047
1048
1049
1050
1051
1052

1053
1054
1055
1056
1057
1058

1059
1060
1061
1062
1063

1064
1065
1066
1067
1068
1069

1070
1071
1072
1073
1074
1075

1076
1077
1078
1079

Metal,PercentageGlod,30Platinum,10Copper,35Silver,25InstructionRedraw the chart with python matplotlib, convert it into a bar chart, giving the code to reflect the fact that the price of \'Gold\' has been reduced to 27% and the \'Silver\' has been increased to 28%. Please keep the colors and legend according to the input chart.Chartimport matplotlib.pyplot as plt# Define the labels and values for the bar chartlabels = ['Silver', 'Gold', 'Copper', 'Platinum']values = [28, 27, 35, 10]colors = ['#7570b3', '#1b9e77', '#e6ab02', '#666666'] # Create a bar chartbars = plt.bar(labels, values, color=colors)# Add labels and title to the chartplt.xlabel('Metal') plt.ylabel('Percentage')plt.title('Metal Prices Comparison in Global Market')# Add a legend to the chartplt.legend(bars, ['Silver', 'Gold', 'Copper', 'Platinum']) # Display the chartplt.show()ChartMoEimport matplotlib.pyplot as plt # 假定的颜色（请根据您的图表颜色进行修改）colors = {'Silver': 'purple', 'Gold': 'gold', 'Copper': 'green', 'Platinum': 'gray'} # 更新后的比例new_values = {'Silver': 28, 'Gold': 27, 'Copper': 35, 'Platinum': 10} # 计算总和以便将值转换为百分比total_value = sum(new_values.values())percentages = {metal: (value / total_value * 100) for metal, value in new_values.items()} # 创建条形图fig, ax = plt.subplots() # 绘制条形图for metal, percentage in percentages.items():    ax.barh(metal, percentage, color=colors[metal]) # 设置X轴标签ax.set_xlabel('Percentage (%)') # 设置图表标题ax.set_title('Metal Prices Comparison in Global Market') # 显示图表plt.show()Qwen-2.5import matplotlib.pyplot as plt # Datametals = ['Gold', 'Silver', 'Copper', 'Platinum']prices = [27, 28, 35, 10]colors = ['mediumseagreen', 'mediumpurple', 'gold', 'gray'] # Plotplt.figure(figsize=(10, 6))plt.bar(metals, prices, color=colors) # Labels and titleplt.title('Metal Prices Comparison in Global Market')plt.xlabel('Metals')plt.ylabel('Price Percentage') # Legendplt.legend(metals) # Show plotplt.show() GPT-4OTable:Under review as a conference paper at ICLR 2025

1080
1081
1082
1083

1084
1085
1086
1087
1088
1089

1090
1091
1092
1093
1094

1095
1096
1097
1098
1099
1100

1101
1102
1103
1104
1105
1106

1107
1108
1109
1110
1111
1112

1113
1114
1115
1116
1117

1118
1119
1120
1121
1122
1123

1124
1125
1126
1127
1128
1129

1130
1131
1132
1133

Figure 15: Chat demo involves modifying chart editing. The bar chart is without labeled data points, and all
methods provide reasonable and executable drawing code. Qwen-2.5 directly identifies the highest element,
while the other two methods make it by code. Note that ChartMoE delivers the most accurate values, thanks to
extensive alignment training and proposed MoE architecture.

21

import numpy as npimport matplotlib.pyplot as plt # Data from the chartyears = ['2016', '2017', '2018', '2019', '2020', '2021', '2022']gpas = [3.2, 3.4, 3.6, 4.0, 3.8, 3.9, 4.0]highest_gpa_year = '2019' # Find the index of the year with the highest GPAhighest_gpa_index = years.index(highest_gpa_year) # Plotting the chartfig, ax = plt.subplots() ax.bar(years[:-1], gpas[:-1], color='blue')ax.bar([highest_gpa_year], [gpas[highest_gpa_index]], color='red') plt.xlabel('Year')plt.ylabel('Student A Average GPA')plt.title('Student Performance') # Adding the legendax.legend(['Student A Average GPA'], loc='upper left') # Show the plotplt.show()  import matplotlib.pyplot as plt # datadata = [3.3, 3.5, 3.6, 3.8, 3.7, 3.6, 3.8]years = ['2016', '2017', '2018', '2019', '2020', '2021', '2022']labels = ['Student A Average GPA']colors = ['blue'] # plotting the chartplt.bar(years, data, color=colors)plt.title('Student Performance')plt.xlabel('Year')plt.ylabel('Student A Average GPA')plt.legend(labels) # get the highest gpahighest_score_index = data.index(max(data))plt.bar(years[highest_score_index], data[highest_score_index], color='red') plt.show()import matplotlib.pyplot as pltimport numpy as np # Datayears = np.array([2016, 2017, 2018, 2019, 2020, 2021, 2022])gpas = np.array([3.2, 3.5, 3.8, 3.9, 3.7, 3.8, 4.0]) # Find the year with the highest GPAmax_index = np.argmax(gpas) # Colors: blue for the bars, red for the highest scorecolors = ['blue' if i != max_index else 'red' for i in range(len(years))] # Plotplt.figure(figsize=(10, 6))bars = plt.bar(years, gpas, color=colors, label='Student A') # Highlight the year with the highest scorebars[max_index].set_color('red') # Labels and titleplt.xlabel('Year')plt.ylabel('Grade Point Average GPA')plt.title('Student Performance')plt.legend() # Show the plotplt.tight_layout()plt.show() Year,GPA2016,3.32017,3.52018,3.62019,3.82020,3.72021,3.62022,3.8 Redraw the chart with python matplotlib, giving the code to highlight the column corresponding to the year in which the student got the highest score (painting it red). Please keep the same colors and legend as the input chart.ChartMoEQwen-2.5GPT-4OChartInstructionTableUnder review as a conference paper at ICLR 2025

1134
1135
1136
1137

1138
1139
1140
1141
1142
1143

1144
1145
1146
1147
1148

1149
1150
1151
1152
1153
1154

1155
1156
1157
1158
1159
1160

1161
1162
1163
1164
1165
1166

1167
1168
1169
1170
1171

1172
1173
1174
1175
1176
1177

1178
1179
1180
1181
1182
1183

1184
1185
1186
1187

Figure 16: Chat demo involves modifying chart editing. The line chart is without labeled data points, and
all methods provide reasonable and executable drawing code. The values extracted by all models differ from
the ground truth, but both ChartMoE and GPT-4O captured the correct data trends. Additionally, ChartMoE
successfully completed all the editing tasks specified in the instructions.

22

import numpy as npimport matplotlib.pyplot as plt# 假设数据years = np.array([2010, 2012, 2014, 2016, 2018, 2020, 2022])growth_rates = np.array([-1.5, 0.5, -2.5, 1.5, -3.5, 4.5])# 找出最低增长率的索引min_growth_rate_idx = np.argmin(growth_rates)# 重新绘制图表fig, ax = plt.subplots()ax.plot(years, growth_rates, marker='o', linestyle='--', \        color='blue', label='Growth Rate in Percentage')ax.axhline(y=growth_rates[min_growth_rate_idx], xmin=0.1, \   xmax=0.9, dashes=[5, 5], color='gray', alpha=0.5, zorder=-1) ax.annotate('lowest', xy=(years[min_growth_rate_idx], \   growth_rates[min_growth_rate_idx]), xytext=(-10, -10), \   textcoords='offset points', ha='right', va='bottom', \   bbox=dict(boxstyle='round', fc='white'))ax.set_xlabel('Years')ax.set_ylabel('Growth Rate in Percentage')ax.set_title('Global GDP Growth')ax.set_xlim(min(years)-1, max(years)+1)ax.set_ylim(min(growth_rates)*1.1, max(growth_rates)*1.1)ax.grid(True)# 设置图例位置legend = ax.legend(loc='lower right', shadow=True)legend.get_frame().set_facecolor('white')plt.show() import matplotlib.pyplot as plt import pandas as pd # Data: years and corresponding growth ratesdata = {'Years': [2010, 2012, 2014, 2016, 2018, 2020, 2022], 'Growth Rate in Percentage': [3.5, 4.3, 3.8, 3.1, 3.2, -4, 5.2]} df = pd.DataFrame(data) # Plot the growth rate over the yearsplt.plot(df['Years'], df['Growth Rate in Percentage'], \     label='Growth Rate in Percentage') # Define the year and value for the lowest pointlowest_point = 2020lowest_value = -4 # Add a dotted horizontal line for the lowest pointplt.axhline(lowest_value, color='gray', linestyle='dotted', label='Lowest')plt.text(lowest_point, lowest_value - 0.5, 'Lowest', color='gray') # Show legend and display the chartplt.legend()plt.show()# Let's use matplotlib to create a similar chart, and incorporate the requested changes.import matplotlib.pyplot as plt # Data for global GDP growth rateyears = [2010, 2012, 2014, 2016, 2018, 2020, 2022]# Sample data similar to the graph in the imagegrowth_rate = [3.2, 4.1, 4.5, 3.7, 3.3, -4.3, 5.5]  # Create the plotplt.figure(figsize=(14,8))plt.plot(years, growth_rate, label='Growth Rate in Percentage', color='blue', marker='o') # Highlight the lowest data pointmin_growth = min(growth_rate)min_year = years[growth_rate.index(min_growth)] # Draw a dotted horizontal line at the lowest growth rateplt.axhline(y=min_growth, color='red', linestyle='dotted', label='lowest') # Add chart title and labelsplt.title('Global GDP Growth', fontsize=14)plt.xlabel('Years', fontsize=12)plt.ylabel('Growth Rate in Percentage', fontsize=12) # Add the legend and place it with an anchorplt.legend(loc='upper right', fontsize=10) # Display the plotplt.show()  Year,GDP Growth2010,3.52012,4.32014,3.82016,2.92028,3.22020,-4.42022,5.2 Redraw the chart with python matplotlib, giving the code to highlight data point with lowest growth rate (draw a horizontal dotted line parallel to the x-axi, through the lowest point and add \'lowest\' label in the legend anchor). Please keep the same colors and legend as the input chart.ChartMoEQwen-2.5GPT-4OChartTableInstruction