MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine
Under review as a conference paper at ICLR 2025

MAVIS:
MATHEMATICAL VISUAL INSTRUCTION TUNING
WITH AN AUTOMATIC DATA ENGINE

Anonymous authors
Paper under double-blind review

ABSTRACT

Multi-modal Large Language Models (MLLMs) have recently showcased superior
proficiency in general visual scenarios. However, we identify their mathemati-
cal capabilities remain under-explored with three areas to be improved: visual
encoding of math diagrams, diagram-language alignment, and chain-of-thought
(CoT) reasoning. This draws forth an urgent demand for an effective training
paradigm and a large-scale, comprehensive dataset with detailed CoT rationales,
which is challenging to collect and costly to annotate manually. To tackle this
issue, we propose MAVIS, a MAthematical VISual instruction tuning pipeline
for MLLMs, featuring an automatic data engine to efficiently create mathematical
visual datasets. We design the data generation process to be entirely independent
of human intervention or GPT API usage, while ensuring the diagram-caption
correspondence, question-answer correctness, and CoT reasoning quality. With
this approach, we curate two datasets, MAVIS-Caption (558K diagram-caption
pairs) and MAVIS-Instruct (834K visual math problems with CoT rationales), and
propose four progressive stages for training MLLMs from scratch. First, we utilize
MAVIS-Caption to fine-tune a math-specific vision encoder (CLIP-Math) through
contrastive learning, tailored for improved diagram visual encoding. Second, we
also leverage MAVIS-Caption to align the CLIP-Math with a large language model
(LLM) by a projection layer, enhancing vision-language alignment in mathematical
domains. Third, we adopt MAVIS-Instruct to perform the instruction tuning for
robust problem-solving skills, and term the resulting model as MAVIS-7B. Fourth,
we apply Direct Preference Optimization (DPO) to enhance the CoT capabilities
of our model, further refining its step-wise reasoning performance. On various
mathematical benchmarks, our MAVIS-7B achieves leading results among open-
source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best
LLaVA-NeXT (110B) by +6.9%, demonstrating the effectiveness of our method.

1

INTRODUCTION

The pursuit of artificial general intelligence necessitates models to seamlessly interpret and generate
multi-modal data. In recent years, the advent of Large-language Models (LLMs) (Brown et al., 2020;
Touvron et al., 2023a;b; Chiang et al., 2023) and their Multi-modal extension (MLLMs) (Zhang
et al., 2024a; Gao et al., 2023b; Su et al., 2023; Ye et al., 2023a) have significantly facilitated this
process across various fields, such as healthcare (Singhal et al., 2023; Shu et al., 2023), autonomous
driving (Yang et al., 2023; Jin et al., 2024), and robotics (Li et al., 2023b; Liu et al., 2024b). Although
MLLMs exhibit remarkable performance in diverse tasks and benchmarks, one arena where they
have yet to fully demonstrate their potential is mathematical problem-solving in visual contexts.

Existing efforts (OpenAI, 2023b;a; Zhou et al., 2023) for text-only mathematics have attained
considerable progress, largely attributed to the availability of sufficient and easily accessible training
data. In contrast, solving visual mathematical problems remains a significant challenge for MLLMs,
primarily due to the absence of a fully validated, effective training pipeline and the acute shortage of
large-scale, high-quality datasets. Visual mathematical data is not only more costly to collect from
publicly available sources compared to text-only data, but also requires expensive manual annotation
to produce accurate step-by-step chain-of-thought (CoT) rationales integrating diagram information.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: (a) We compare the attention map of class tokens from CLIP ViT-L (Radford et al., 2021)
and our CLIP-Math. Our vision encoder can better capture significant mathematical information
within diagrams. (b) We compare the diagram captioning capabilities between GPT-4V (OpenAI,
2023c) and our MAVIS-7B, where GPT-4V fall short of accurately recognizing mathematical elements.
(c) We compare the chain-of-thought (CoT) reasoning between different models, showcasing that
GPT-4V and Gemini-Pro (Gemini Team, 2023) suffer from low-quality reasoning process.

In light of these challenges, we identify three critical issues that impede the visual mathematical
capabilities of MLLMs.

i. Unsatisfactory math diagram embeddings by vision encoders. Most MLLMs adopt a
frozen CLIP (Radford et al., 2021) as the vision encoder, which is pre-trained by natural
images capturing real-world scenes with rich colors and textures. In contrast, math diagrams
are composed of abstract curves, shapes, and symbols with a monochromatic color scheme,
exhibiting large semantic gaps to general scenarios. As visualized in Figure 1 (a), the
attention map of CLIP struggles to capture important information within math diagrams,
which cannot provide satisfactory visual embeddings for LLMs to understand.

ii. Diagram-language misalignment between vision encoders and LLMs. Likewise, the
vision-language pre-training stage of MLLMs also adopts natural image-caption pairs for
cross-modal alignment. Due to the domain gap, while they can generate accurate captions for
real-world images, but fall short of recognizing basic mathematical elements and narrating
their relations. As compared in Figure 1 (b), even GPT-4V (OpenAI, 2023c) produces
low-quality descriptions for simple geometric figures and functions, indicating LLMs are
not well aligned with the visual embedding space of math diagrams.

iii. Inaccurate CoT reasoning capabilities with visual elements by MLLMs. Referring to
the CoT evaluation in MathVerse (Zhang et al., 2024b), incorporating the diagram input

2

Findthedomainandrangeofthefunctionfusingintervalnotation,whichgoesthrough(-2,-4),(-1,-2),(0,0),(1,-4).Question：Answer：domain(-3,1]     range[-4,0]GPT-4V:Fromthegraph,Thefunctionstartsatx=−4andgoesuptox=1.Thereisahollowpointatx=−3,whichmeansthefunctionisnotdefinedthere.Thus,thedomainis[-4,-3)∪(-3,1]Therangeisthesetofallpossibleoutputvalues(y-values).Fromthegraph:Thelowesty-valueis−4(atx=−2andx=1).Thehighesty-valueis0,butsincethepointat(−3,0)ishollow,thefunctiondoesnotreach0.Thus,therangeis[-4,0)Gemini-Pro:Thedomainofthefunctionisthesetofallx-valuesforwhichthefunctionisdefined.Thefunctionisdefinedforallx-valuesbetween-3and1,including-3butexcluding1.Thus,Wecanwritethisinintervalnotationas:[-3,1)Therangeofthefunctionisthesetofally-valuesthatthefunctioncantakeon.Thefunctiontakesonally-valuesbetween-4and0,including-4and0.Wecanwritethisinintervalnotationas:[-4,0]MAVIS-7B:Fromthegraph,weknowthatthefunctionstartsfromx=-3andendsatx=1.Besides,thereisahallowpointatx=-3.Thus,Wecandeterminethatthedomainis(-3,1]Fromthegraph,weknowthatthefunctionreachesmaximum0atx=0,andreachesminimus-4atx=-2andx=1.Thefunctionattainsallvaluesfrom-4to0.Thus,therangeis[-4,0]××××××✓✓✓✓✓✓✓✓✓✓✓✓✓TheimageshowsageometricfigurewithpointslabeledA,B,C,D,andE.PointAisatthetopofaverticallinesegmentAB,whichmeasures10units.FromB,a30°angleisformedwiththelinesegmentextendingtoD.ThereisaquartercirclecenteredatBwitharadiusof10units,reachingpointC.ThefigureincludesaninscribedcirclecenteredwithinthequadrilateralformedbypointsC,D,andE.ThelinesABandDEareparallel...AsectorABCisdrawn,withangleCBAmeasuring30degrees.AsquareCBDEislinkedtoedgeCB,withangleBCEmeasuring90degrees.ThedistancebetweenpointsAandBis10.0units,andthedistancebetweenpointsCandBisalso10.0units.ThesquareCBDEhasaninscribedcircle.Math DiagramCLIP ViT-LOur CLIP-MathGPT-4V:MAVIS-7B:(a)  Attention Comparison betweenCLIP and our CLIP-Math (b)  Captioning Comparison betweenGPT-4V and our MAVIS-7B(c)  CoTReasoning Comparison between Different MLLMsUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

would adversely affect the reasoning quality of MLLMs compared to using only the text-
only question. As visualized in Figure 1 (c), we observe the problem-solving process of
GPT-4V and Gemini-Pro (Gemini Team, 2023) both suffer from low-quality CoT reasoning
accuracy. This demonstrates the incapability of MLLMs to leverage visual cues for precise
step-by-step mathematical problem-solving.

Therefore, to mitigate these issues, it is essential to develop an extensive dataset and effective training
approach tailored to visual mathematics. In this paper, we propose MAVIS, a MAthematical VISual
instruction tuning paradigm and an automatic data generation engine for MLLMs, which aims to fully
unleash their potential for diagram visual encoding and reasoning capabilities. We introduce two
meticulously curated datasets, a progressive four-stage training pipeline, and a visual mathematical
specialist, MAVIS-7B. We summarize the contributions of our work as follows.

• Automatic Mathematical Visual Data Engine. To eliminate the need for labor-intensive
annotation and expensive GPT API (OpenAI, 2023c;b) usage, we designed our data engine
to be entirely rule-based and fully automated. This engine handles every aspect of math-
ematical data creation, including diagram drawing, caption generation, question-answer
synthesis, and CoT rationale production. With this approach, we curate two large-scale,
high-quality mathematical visual datasets, MAVIS-Caption and MAVIS-Instruct, widely
covering plane geometry, analytic geometry, and function. MAVIS-Caption consists of
558K diagram-caption pairs automatically created by our data engine with accurate vision-
language correspondence. MAVIS-Instruct includes 834K visual math problems, which
includes 582K data constructed by our data engine and additional 252K data augmented by
GPT-4V from manual collection and existing datasets (Chen et al., 2021c; Lu et al., 2021).
Each problem is annotated with a CoT rationale, and modified to contain minimized textual
redundancy that enforces MLLMs to pay more attention on visual diagrams.

• Four-stage Training Pipeline. Our training framework involves four progressive stages
designed to sequentially address the aforementioned identified deficiencies in MLLMs.
Firstly, we utilize MAVIS-Caption to fine-tune a math-specific vision encoder by contrastive
learning, termed CLIP-Math, to enable better visual representations of math diagrams.
Subsequently, we align this encoder with the LLM to ensure effective diagram-language
integration also by MAVIS-Caption. After that, our MAVIS-Instruct is adopted to instruction-
tune the MLLM, which provides sufficient step-wise problem-solving supervision. Finally,
we employ Direct Preference Optimization (DPO) (Rafailov et al., 2024) with annotated CoT
rationales in MAVIS-Instruct to further enhance the reasoning capabilities of our model.
• Mathematical Visual Specialist. After the four-stage training, we develop MAVIS-7B,
an MLLM specifically optimized for visual mathematical problem-solving. On various
evaluation benchmarks, our model achieves leading performance compared to existing
open-source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best
LLaVA-NeXT (110B) (Li et al., 2024a) by +6.9% on MathVerse (Zhang et al., 2024b). The
quantitative results and qualitative analysis both validate the significance of our approach.

2 AUTOMATIC DATA ENGINE

To cope with the substantial data requirements of MLLMs, it is essential to have access to extensive
training instances. However, for visual mathematics, the paucity of publicly available datasets poses
a challenge, and creating such data manually also involves a high cost. Therefore, as illustrated in
Figure 2, we develop an automatic data engine to efficiently generate high-quality math diagrams
(Section 2.1), captions (Section 2.2), and question-answer with rationales (Section 2.3).

2.1 DIAGRAM GENERATION

Covering most mathematical scenarios, we adopt three diagram types: plane geometry, analytic
geometry, and function. Note that all the logic of the data engine is implemented in Python, and we
employ Matplotlib for the graphical rendering of the diagrams.

Plane Geometry Diagram. As such diagrams typically consist of spatial combinations of various
basic shapes, we utilize principles from multi-hop data curation to develop customized generation

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 2: Overview of Automatic Data Engine. We present the generation pipelines of geometry
(Top) and function (Bottom) problems within the proposed automatic data engine, including diagrams,
questions, captions, and Chain-of-Thought (CoT) rationales.

rules. These rules allow for the iterative integration of new shapes into existing configurations.
Initially, we establish a core set of shapes, including squares, rectangles, triangles, sectors, etc, for
diagram generation. Starting with a randomly selected shape, we extend another shape from the set
along one of its straight sides. By iterating this process, we can construct diverse plane geometry
diagrams featuring different combinations of shapes. Additionally, we randomly label the vertices
with letters (e.g., A, B, C) and annotate numerical values relevant to geometric properties (e.g., side
lengths and angles), simulating realistic plane geometry problems.

Analytic Geometry Diagram. Likewise, our approach begins by defining a basic figure set that
differs slightly from that used in plane geometry; for example, we include additional elements such
as points and line segments. We then construct a Cartesian coordinate system, complete with grid
lines and scaled axes. The range of the coordinate system is randomly determined within a predefined
scope. Subsequently, we select a number from 1 to 3 to indicate the number of figures to be drawn on
the graph, and randomly choose coordinates for the top-left vertices to plot these figures at varied
sizes (using these points as centers for circles). Unlike plane geometry, we ensure that the figures do
not overlap, except for points and segments, and maintain the figure areas within a suitable scale.

Function Diagram. We focus on seven fundamental function types: polynomial, sine, cosine,
tangent, logarithmic, absolute value, and piece-wise polynomial functions. For each function type,
we parameterize the equations with random variables, such as coefficients and constants within a
predefined range (e.g., a and b in y = ax + b), which facilitates the generation of diverse function
graphs. We also adopt the same Cartesian coordinate system employed for analytic geometry.
Additionally, for specific caption or question-answering samples, we also plot key features like
extreme points and zero points of the functions, providing additional visual information that aids in
the understanding and reasoning of these mathematical functions.

2.2 MAVIS-CAPTION

With our mathematical visual data engine, we first curate a diagram-caption dataset, MAVIS-Caption,
as shown in Figure 3, aiming to benefit the diagram visual representations and cross-modal alignment.

Data Overview. As presented in Table 3 of the Appendix, the MAVIS-Caption dataset comprises
588K diagram-caption pairs. This includes 299K for plane geometry, 77K for analytic geometry, and

4

O(3,	-2log(16))xyOxyRationaleFundamental ShapesABCABCisanisoscelestriangle,whereAB=AC.AB=2.AB = 2,∠A = 30°ABC2SinceAB = 2and∠A=30°,wecancalculatetheangleCofisoscelestriangleas∠C=(180-30)/2=75°.Using the Law of Sines,BC=sin(∠A)*AB/sin(∠C)=0.5*2/0.97=1.04.2ABCD2ExtendedfromsideBC,BCDconformsasector.∠CBD=90°.ABCD2TheareaofsectorCBD=r*r*π/4=0.85.BC= 1.04, ∠CBD = 90°CaptionConditionRationaleConditionCaptionSelect arandom edgeExtend arandom shapeSelect an attributeto calculateSelect arandom shapeCaptiontemplateRationale templateCaptiontemplateRationale templateRationaleSetconditions andquestionsSolveExpressionSelect arandom functionRationale templateThefigureshowsthegraphoff(x)=-2*log(5*x+1).f(x) = -2*log(5*x + 1)ExpressionCaptionCaptiontemplatea= -2Conditionx= 2, y’ = -10/11 x= 3, y= -2*log(16)What is the area of sector CBD?QuestionWhat arethezeros?QuestionSolveAttributesRationaleRationale templateConsideringx = 3 is... -2*log(3*b+ c) = -2*log(16), and... 3*b+ c = 16;Differentiating... f'(x) = -2*b/(b*x + c)... whenx = 2 is -10/11, -2*b= -20*b/11 -10*c/11;Fromsolvingtheequations, wefindthatb= 5 andc = 1...f(x) = -2*log(5*x + 1).Let-2*log(5*x + 1) = 0. Simplificationof theequationleadsto5*x + 1 = 1.The solutiontotheequationsis x=0.Asa result, wecanconcludethatthezeroof thefunctionis0.Fundamental FunctionsGeometry Problem Generation:Function Problem Generation:Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 3: MAVIS-Caption Dataset. We showcase three diagram-caption pairs of plane geometry,
function, and analytic geometry in MAVIS-Caption, generated by our developed data engine.

212K for function. The average word length of the captions is 61.48 words, reflecting their detailed
descriptive nature. The overall vocabulary size is 149, indicating the diversity in language expression.
We adopt different strategies to generate captions for three types of diagrams. It is important to note
that GPT-4 (OpenAI, 2023b) is only utilized during the template creation stage; it is not used at any
point during the automatic caption generation process.

Plane Geometry Caption. We follow the iterative geometric generation process to develop regula-
tions for an accurate and detailed caption. We first prompt GPT-4 to create three sets of language
templates: the descriptive content for fundamental shapes (e.g., “A Triangle {} with two congruent
sides {} and {}”), the phrases to denote specific attributes (e.g., “Angle {} measures {} degrees”), and
the conjunction to link two adjacent shapes (e.g., “Attached to edge {} of shape {}, there is a {}”).
Then, based on various generation scenarios, we fill and merge these templates to acquire a coherent
description of the geometric figure.

Function Caption. As function diagrams typically showcase a single curve, we directly utilize GPT-
4 to generate templates describing various properties of functions, including expressions, domains,
ranges, extreme points, and zero points. Each template is then filled based on specific cases, such as
“The expression of the function is y = −3x3 − 2x2 − 2x − 2. Within the range of x values [−3.0, 4.0],
zero points occur at −0.83 ...”.

Analytic Geometry Caption. We also employ GPT-4 to obtain two sets of language templates: the
description of coordinates and attribute information for basic figures (e.g., “The square with its base
left corner at {} features sides of {} in length”) and the spatial relation for nearby figures (e.g., “On
the bottom right of {}, there is a {}”). The captions are then formulated by filling in the coordinates
and selecting appropriate spatial relationship templates through coordinate comparison.

2.3 MAVIS-INSTRUCT

Besides the diagram-caption data, we curate MAVIS-Instruct of extensive problem-solving data,
which endows MLLMs with visual mathematical reasoning capabilities and serve as the basis for
Direct Preference Optimization (DPO) (Rafailov et al., 2024), as shown in Figure 5.

Data Overview. As illustrated in Table 4 of the Appendix, the MAVIS-Instruct dataset consists of
a total of 834K visual math problems. Given that the proportion of analytic geometry problems is
relatively small, we classify them with function problems for simplicity. Each problem in MAVIS-
Instruct includes a CoT rationale providing step-by-step solutions, with an average answer length
of 150 words. We have minimized textual redundancy in the questions, eliminating unnecessary
contextual information, distracting conditions, and attributes readily observable from the diagrams.

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 4: MAVIS-Instruct Dataset. We showcase the generated visual math problems from four
sources within MAVIS-Instruct, which contain detailed rationales and minimized textual redundancy.

This reduction in text forces MLLMs to enhance their capability to extract essential content from
visual inputs. MAVIS-Instruct is assembled from four distinct sources to ensure broad coverage.

Data Engine Generated Problems. Within our data engine, we manually craft rigorous regulations
to produce visual math problems with accurate CoT annotations. Similar to caption generation, GPT
API is not involved in the automatic synthesis process of questions, answers, and CoT rationales.

• Plane Geometry Problems. We initially prompt GPT-4 to compile a comprehensive set of
mathematical formulas applicable to each basic shape (e.g., Pythagorean theorem for right
triangles and area formula for circles). Then, for a geometric diagram, we randomly select
a known condition within a shape as the final solution target, and systematically deduce
backward to another condition, either within the same shape or an adjacent one, using a
randomly selected mathematical formula. This deduced condition is then set as unknown,
and we continue iterative backward deductions as necessary. The final condition, along
with any conditions in the last step, are presented as initial attributes in the question. The
rationales can be simply obtained by reversing this backward deduction process.

• Function Problems. As the properties of functions are predetermined, we utilize GPT-4 to
generate diverse reasoning templates. These templates facilitate the solving of one function
property based on other provided properties, thereby ensuring the generation of high-quality
function rationales. The related function properties include analytical expression, function
values, zeros, extremum points, monotonicity, derivatives, and integrals. To accurately
reason these properties, the CoT annotation incorporates understanding of function types,
solving the analytical expressions of equations, and interpreting function graphs.

Data Engine Captions Annotated by GPT-4. Given the detailed captions and diagrams generated
by our data engine, we can prompt GPT-4V with these sufficient conditions to synthesis question-
answering data and ensure its correctness. We first generate a new set of 17K diagram-caption pairs
that do not overlap with the previous MAVIS-Caption, which avoids answer leakage within the
detailed caption. Then, we prompt GPT-4V to generate 3 new problems with rationales, obtaining
51K data in total from the diagram-caption pairs.

Manual Collection Augmented by GPT-4. To incorporate high-quality problems found in real-
world contexts, we manually collect 4K math problems with diagrams from publicly available
resources. Recognizing that these sources often lack detailed rationales and may contain redundant

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Figure 5: Four-stage Training Pipeline of MAVIS. With our curated MAVIS-Caption and MAVIS-
Instruct, we adopt four progressive stages for training a mathematical visual specialist from scratch.

text, we initially utilize GPT-4V to annotate a detailed solving process and streamline the question
text to reduce redundancy. Subsequently, for each collected instance, we input the question, rationale,
and diagram into GPT-4 and employ customized few-shot prompts to generate 20 new problems per
original, comprising 15 multiple-choice questions and 5 free-form questions. This process contributes
a total of 83K problems to the dataset.

Existing Datasets Augmented by GPT-4. Given existing well-organized geometric datasets, we
can also leverage them to expand MAVIS-Instruct. Referring to previous prompt designs, we augment
the 8K training set from two dataset, Geometry-3K (Lu et al., 2021) and GeoQA+ (Chen et al.,
2021b), into 80K visual problems with accompanying rationales, mapping each original problem
to 10 new ones. Due to the scarcity of publicly available function data, we do not include function
problems from this source.

3 MATHEMATICAL VISUAL TRAINING

With the curated datasets, we devise a four-stage training pipeline for endowing MLLMs with
mathematical visual capabilities. They respectively aim to mitigate the three deficiencies within
existing MLLMs, i.e., diagram visual encoding, diagram-language alignment, and mathematical
reasoning skills in visual contexts.

3.1 STAGE 1: TRAINING CLIP-MATH

To enhance CLIP’s (Radford et al., 2021) inadequate visual encoding of math diagrams, we utilize
MAVIS-Caption to train a specialized CLIP-Math encoder. Specifically, we fine-tune a pre-trained
CLIP-Base model following the conservative learning scheme. The math diagrams are fed into
the learnable vision encoder, while the corresponding captions are processed by the text encoder,
which remains frozen to provide reliable supervision. Via contrastive training, the model learns
to adapt from its original natural image domain to mathematical contexts, increasing its focus on
essential visual elements within diagrams, as demonstrated in Figure 1 (a). The optimized CLIP-Math
encoder now delivers more precise and robust representations of math diagrams, establishing a solid
foundation for the subsequent visual interpretation of LLMs.

3.2 STAGE 2: ALIGNING DIAGRAM-LANGUAGE

After acquiring the CLIP-Math encoder, we further integrate it with LLMs using MAVIS-Caption
to boost cross-modal alignment between math diagrams and language embedding space. Using a
simple two-layer MLP as the projection layer, we transform the visual encodings from CLIP-Math,
and prepend them as a prefix to the LLM input. This process, guided by the diagram captioning
task, enables the LLM to accurately recognize mathematical components and spatial arrangements.
With the diagram-language alignment, LLMs are equipped with the interpretation capability in math
diagrams, serving as an initial step toward deeper mathematical reasoning. In this stage, we freeze
the CLIP-Math, and train the projection layer along with the LoRA-based (Hu et al., 2021) LLM.

3.3 STAGE 3: INSTRUCTION TUNING

On top of that, we leverage MAVIS-Instruct to endow MLLMs with CoT reasoning and problem-
solving capabilities in visual mathematics. The detailed rationales within each problem’s solution
provide high-quality reasoning guidance for MLLMs, significantly enhancing their step-by-step

7

Training CLIP-MATHAligning Diagram-LanguageInstruction TuningPreference Alignmentwith DPOStage-1Stage-2Stage-3Stage-4Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 1: Evaluation on MathVerse’s testmini Set with Six Problem Versions. ‘CoT-E’ and ‘Acc’
denote the scores of CoT evaluation strategy and the scores of direct ‘true or false’ accuracy, respec-
tively. ‘∗’ denotes previous mathematical visual specialists. The highest scores for closed-source
and open-source MLLMs are marked in red and blue respectively.

Model

All

LLM
Size

Text
Dominant

Text
Lite

Vision
Intensive

Vision
Dominant

Vision
Only

CoT-E

Acc

CoT-E

Acc

CoT-E

Acc

CoT-E

Acc

CoT-E

Acc

CoT-E

Acc

Random Chance
Human

ChatGPT
GPT-4

Qwen-VL-Plus
Gemini-Pro
Qwen-VL-Max
GPT-4V

-
-

-
-

-
-
-
-

LLaMA-Adapter-V2
ImageBind-LLM
mPLUG-Owl2
MiniGPT-v2
LLaVA-1.5
SPHINX-Plus
G-LLaVA∗
LLaVA-NeXT
ShareGPT4V
SPHINX-MoE
Math-LLaVA∗
InternLM-XC2.
LLaVA-NeXT

MAVIS-7B w/o DPO∗
MAVIS-7B∗

7B
7B
7B
7B
7B
13B
7B
8B
13B
8×7B
13B
7B
110B

7B
7B

-
-

-
-

21.3
35.3
37.2
54.4

5.8
10.0
10.3
10.9
12.7
14.0
15.7
17.2
17.4
22.8
24.1
25.9
28.3

33.7
35.2

12.4
64.9

-
-

-
-

11.8
23.5
25.3
39.4

5.7
9.2
5.9
11.0
7.6
12.2
16.6
15.6
13.1
15.0
19.0
16.5
24.5

27.5
28.4

51.3
63.4

26.0
39.8
42.8
63.1

7.8
13.2
11.6
13.2
17.1
16.3
22.2
21.6
21.8
33.3
34.2
36.9
37.1

42.5
43.2

Baselines

12.4
71.2

33.3
46.5

-
-

LLMs

38.5
40.7

12.4
70.9

18.9
20.7

Closed-source MLLMs

15.7
26.3
30.7
54.7

21.2
34.7
37.7
56.6

Open-source MLLMs

6.2
11.4
6.6
12.1
8.8
13.9
20.9
19.4
16.2
22.2
21.2
22.3
31.7

41.4
41.6

6.3
11.6
11.4
12.7
12.0
12.8
20.4
19.7
20.6
21.9
22.7
28.3
29.1

36.3
37.2

11.1
23.5
26.1
41.4

5.9
11.3
6.3
12.0
7.6
11.6
20.7
15.2
16.2
16.4
19.8
17.0
24.1

29.1
29.5

-
-

-
-

18.5
32.0
33.6
51.4

6.2
9.8
11.1
11.1
12.6
12.9
16.5
17.6
18.6
21.1
21.1
20.1
22.6

33.3
34.1

12.4
61.4

-
-

9.0
23.0
24.1
34.9

6.1
8.9
6.3
13.1
7.4
11.6
17.2
16.8
15.5
14.8
20.2
15.7
21.0

27.4
27.9

-
-

-
-

19.1
36.8
35.9
50.8

4.5
11.8
9.4
11.3
12.7
14.7
12.7
14.9
16.2
19.6
20.3
24.4
21.8

29.3
29.7

12.4
68.3

-
-

13.0
22.3
24.1
34.4

4.2
11.2
5.6
10.3
7.4
13.5
14.6
15.2
13.8
12.6
17.6
16.4
22.1

24.9
24.7

-
-

-
-

21.8
33.3
35.9
50.3

4.4
3.5
8.0
6.4
9.0
13.2
6.6
12.1
9.7
18.3
22.2
19.8
30.9

27.1
31.8

12.4
66.7

-
-

10.0
22.2
21.4
31.6

6.1
3.4
4.9
7.4
6.9
10.4
9.4
11.3
3.7
9.1
16.4
11.0
20.7

14.6
18.3

CoT process. Furthermore, as we have minimized the redundancy within question texts during the
construction process, such text-lite problem formats, referring to MathVerse (Zhang et al., 2024b),
facilitate MLLMs to capture more essential information from the visual embeddings for problem-
solving, rather than relying on shortcuts to only process the textual content. During this stage, we
unfreeze both the projection layer and the LoRA-based (Hu et al., 2021) LLM to perform a thorough
instruction-following tuning.

3.4 STAGE 4: PREFERENCE ALIGNMENT WITH DPO

After the instruction tuning phase, the resulting model gains the capability for CoT reasoning on
visual math problems. However, it may still produce inaccurate intermediate steps due to insufficient
supervision for generating the best reasoning path. To address this, we further apply CoT preference
alignment using the DPO (Rafailov et al., 2024) algorithm to further enhance the model’s reasoning
performance. Specifically, we adopt the instruction-tuned model to first infer CoT reasoning process
on the 582K problems generated by data engine within MAVIS-Instruct. Then, we filter out the
incorrect outputs (88K data) based on the final answer as the negative reasoning samples in DPO,
and directly utilize the annotated CoT process as the positive samples. We only unfreeze the LoRA
parameters for DPO training, and finally obtain our mathematical specialist, MAVIS-7B.

4 EXPERIMENT

We first detail our experimental settings in Section 4.1, and then discuss the quantitative on different
benchmarks and qualitative examples in Sections 4.2 and 4.3, respectively. Please refer to the
Appendix for more data details and ablation studies.

8

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 2: Evaluation on Six Mathematical Benchmarks. ‘MMMU-Math’ denotes the math problems
within the test set of MMMU. ‘GPS’, ‘ALG’, and ‘GEO’ denote geometry problem solving, algebraic,
and geometry in MathVista’s testmini set. ‘S1’, ‘S2’, and ‘S3’ denote different problem steps in
We-Math’s testmini set. ‘∗’ denotes previous mathematical visual specialists. The highest scores for
closed-source and open-source MLLMs are marked in red and blue respectively.

Model

LLM
Size

GeoQA FunctionQA MMMU-Math MathVision

MathVista
ALG

GPS

GEO

S1

We-Math
S2

S3

Random Chance
Human

ChatGPT
GPT-4

Qwen-VL-Plus
Qwen-VL-Max
GPT-4V

-
-

-
-

-
-
-

LLaMA-Adapter V2
mPLUG-Owl2
UniMath
LLaVA-1.5
ShareGPT4V
SPHINX-MoE
G-LLaVA∗
Math-LLaVA∗
InternLM-XC2.
LLaVA-NeXT
MAVIS-7B w/o DPO∗
MAVIS-7B∗

7B
7B
-
13B
13B
8×7B
13B
13B
7B
110B

7B
7B

17.1
92.3

-
-

-
-
-

18.1
15.7
50.0
20.3
-
-
67.0
62.3
66.4
-

66.7
68.3

Baselines

21.6
84.2

LLMs

-
30.6

7.2
68.8

9.7
13.1

Closed-source MLLMs

-
36.3
48.4

10.7
15.6
22.8

Open-source MLLMs

23.0
18.8
-
24.0
-
-
27.6
36.1
30.1
-

39.2
42.4

8.2
8.6
-
11.2
11.9
14.2
1.3
15.5
14.5
-

18.6
19.2

-
-

-
-

-
-
-

30.6
29.0
-
33.9
-
33.9
24.2
38.7
38.7
-

40.3
50.0

24.1
48.4

31.7
31.7

38.5
-
50.5

25.5
12.5
-
16.3
-
31.2
36.1
57.7
63.0
-

63.2
64.1

25.8
50.9

32.4
33.5

39.1
-
53.0

26.3
27.7
-
38.5
-
31.7
24.6
53.0
56.6
-

58.3
59.2

22.7
51.4

33.0
32.2

39.3
-
51.0

24.3
14.2
-
16.7
-
30.5
33.1
56.5
62.3
-

63.0
63.2

-
-

-
-

-
40.8
65.5

-
-
-
-
-
-
32.4
37.5
47.0
53.7

56.9
57.2

-
-

-
-

-
30.3
49.2

-
-
-
-
-
-
30.1
30.5
33.1
36.9

37.1
37.9

-
-

-
-

-
20.6
38.2

-
-
-
-
-
-
32.7
32.4
33.0
31.5

33.2
34.6

4.1 EXPERIMENTAL SETTINGS

Implementation Details. We adopt a CLIP ViT-L (Radford et al., 2021) as the pre-trained model to
fine-tune our CLIP-Math, and utilize Mammoth2-7B (Yue et al., 2024) as the base LLM to construct
MAVIS-7B. In the first stage, we fine-tune the CLIP for 10 epochs with a batch size 16 and an initial
learning rate 2e−6. In the second stage, we train the diagram-language alignment for 1 epoch with a
batch size 32 and an initial learning rate 2e−6, and adopt LoRA (Hu et al., 2021) with a rank 128. In
the third and fourth stages, we adopt the same training settings as the second one.

Evaluation Schemes. We evaluate our model MAVIS-7B on several popular mathematical bench-
marks, MathVerse (Zhang et al., 2024b), GeoQA (Chen et al., 2021c), FunctionQA (function problems
in MathVista (Lu et al., 2023)), MMMU-Math (the math problems in MMMU (Yue et al., 2023a)),
MathVision (Wang et al., 2024a), three mathematical categories in MathVista, and We-Math (Qiao
et al., 2024). We compare a variety of existing MLLMs, including two mathematical visual special-
ist (Gao et al., 2023a; Shi et al., 2024), two LLMs (OpenAI, 2023a;b), and other general MLLMs (Bai
et al., 2023b; Gao et al., 2023b; Ye et al., 2023b; Liu et al., 2023a; Chen et al., 2023b; Gao et al.,
2024; Dong et al., 2024; Liu et al., 2024a; Chen et al., 2023a; Gao et al., 2024).

4.2 QUANTITATIVE PERFORMANCE

As shown in Table 1 for the MathVerse benchmark, MAVIS-7B achieves the best overall scores
in both CoT evaluation and accuracy among open-source MLLMs with only a 7B model size, and
consistently surpasses the second-best method on different problem versions. Specifically, our model
surpasses the powerful InternLM-XComposer2 (7B) (Dong et al., 2024) by +9.3% and ShareGPT4V
(13B) (Chen et al., 2023b) by +17.8% CoT evaluation scores. Compared to other mathematical
visual specialist, i.e., G-LLaVA (7B) (Gao et al., 2023a) and the concurrent Math-LLaVA (13B) (Shi
et al., 2024), MAVIS-7B exhibits superior problem-solving capabilities with higher CoT evaluation
scores of +19.5% and +11.1%, respectively. In addition, our model is also advantageous to the most
powerful open-source MLLM series, LLaVA-NeXT (Li et al., 2024a), from 8B to 110B model sizes,
demonstrating the math-specific proficiency of MAVIS-7B. Note that, the improvement brought by

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 6: Problem-solving Comparison of MAVIS-7B and GPT-4V.

DPO (our fourth-stage training) is more apparent in CoT evaluation compared to the accuracy scores,
indicating that the preference alignment learning can effectively boost the CoT reasoning capabilities.

Table 2 showcases the performance comparison on six other mathematical benchmarks, where our
model still attains remarkable performance among other MLLMs. In detail, MAVIS-7B outperforms
the closed-source Qwen-VL-Max (Bai et al., 2023a) by +6.1% in MMMU-Math, +3.6% in MathVi-
sion, and around +10% in three subsets of We-Math. Our model even exceeds GPT-4V (OpenAI,
2023b) in the three mathematical categories of MathVista, indicating our problem-solving and rea-
soning proficiency. We also observe that, the enhancement from DPO increases from ‘S1’ to ‘S3’ of
We-Math, which well demonstrates its benefit on math problems with more intricate reasoning steps.

4.3 QUALITATIVE ANALYSIS

In Figure 6, we compare the mathematical problem-solving examples between MAVIS-7B and GPT-
4V (OpenAI, 2023c). As presented, our model not only showcases better accuracy in understanding
the geometric elements, function curves, and coordinate axes in mathematical diagrams, but also
performs higher-quality step-by-step reasoning process for formula substitution and numerical
calculation. This demonstrates the effectiveness of our four-stage training pipeline and automatic
data engine for enhanced diagram understanding and CoT reasoning.

5 CONCLUSION

In this paper, we propose MAVIS, the first mathematical visual instruction tuning paradigm for
MLLMs. We first introduce two high-quality datasets by a delicate data engine, MAVIS-Caption
and MAVIS-Instruct, containing large-scale diagram-language and problem-solving data. Then, we
customize a three-stage training framework to progressively train the math-specific vision encoder, the
diagram-language alignment, and the mathematical reasoning capabilities of MLLMs. The obtained
specialist model, MAVIS-7B, achieves superior performance across different mathematical visual
benchmarks, demonstrating the potential to serve as a new standard for future research.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REPRODUCIBILITY STATEMENT

In Section 4.1, we demonstrate the implementation details of MAVIS-7B, and in the Appendix, we
provide the dataset details for MAVIS-Caption and MAVIS-Instruct. We provide data examples
generated by our automatic data engine in an anonymous link:

https://anonymous.4open.science/r/MAVIS-ICLR-4C0F/

REFERENCES

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel
Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language
model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–
23736, 2022.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe,
Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for
training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.
ArXiv, abs/2308.12966, 2023a.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou,
and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization,
text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023b.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. In Advances in neural information processing systems, pp. 1877–1901, 2020.

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin.
GeoQA: A geometric question answering benchmark towards multimodal numerical reason-
In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the
ing.
Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513–523, Online, August
2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.46. URL
https://aclanthology.org/2021.findings-acl.46.

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin.
Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning.
ArXiv, abs/2105.14517, 2021b. URL https://api.semanticscholar.org/CorpusID:
235253782.

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin.
Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning.
arXiv preprint arXiv:2105.14517, 2021c.

Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Uni-
geo: Unifying geometry logical reasoning via reformulating mathematical expression. ArXiv,
abs/2212.02746, 2022.

Jun Chen, Deyao Zhu1 Xiaoqian Shen1 Xiang Li, Zechun Liu2 Pengchuan Zhang, Raghuraman
Krishnamoorthi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large
language model as a unified interface for vision-language multi-task learning. arXiv preprint
arXiv:2310.09478, 2023a.

Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua
Lin. Sharegpt4v: Improving large multi-modal models with better captions. ArXiv, abs/2311.12793,
2023b. URL https://api.semanticscholar.org/CorpusID:265308687.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi
Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial
multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,
Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna:
An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://lmsys.org/
blog/2023-03-30-vicuna/, March 2023.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang
Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image
composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420,
2024.

Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu
Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation
benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024.

Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong,
Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal
large language model. arXiv preprint arXiv:2312.11370, 2023a.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu,
Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient
visual instruction model. arXiv preprint arXiv:2304.15010, 2023b.

Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie
Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal
large language models. arXiv preprint arXiv:2402.05935, 2024.

Google Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint

arXiv:2312.11805, 2023.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen,
et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint
arXiv:2309.17452, 2023.

Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen,
Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud
with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint
arXiv:2309.00615, 2023.

Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu,
Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint
arXiv:2309.03905, 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09685, 2021.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris
Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand,
Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-
Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le
Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mixtral of experts. Arxiv 2401.04088, 2024.

Bu Jin, Yupeng Zheng, Pengfei Li, Weize Li, Yuhang Zheng, Sujie Hu, Xinyu Liu, Jinwei Zhu, Zhijie
Yan, Haiyang Sun, et al. Tod3cap: Towards 3d dense captioning in outdoor scenes. arXiv preprint
arXiv:2403.19589, 2024.

Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse:
A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241,
2023.

12

Under review as a conference paper at ICLR 2025

Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang,
Ziwei Liu, and Chunyuan Li.
Llava-next: Stronger llms supercharge multimodal ca-
pabilities in the wild, May 2024a. URL https://llava-vl.github.io/blog/
2024-05-10-llava-next-stronger-llms/.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma,
Tackling multi-image, video,
and 3d in large
URL https://llava-vl.github.io/blog/

and Chunyuan Li.
multimodal models,
2024-06-16-llava-next-interleave/.

Llava-next:
June 2024b.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li.
Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024c.
URL https://arxiv.org/abs/2407.07895.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-language understanding and generation. In International Conference on
Machine Learning, pp. 12888–12900. PMLR, 2022.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and
Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023a.

Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming
Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric
robotic manipulation. arXiv preprint arXiv:2312.16217, 2023b.

Zhenwen Liang, Tianyu Yang, Jipeng Zhang, and Xiangliang Zhang. Unimath: A foundational and

multimodal mathematical reasoner. In EMNLP, 2023.

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi
Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for
multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning, 2023a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS,

2023b.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:
//llava-vl.github.io/blog/2024-01-30-llava-next/.

Jiaming Liu, Chenxuan Li, Guanqun Wang, Lily Lee, Kaichen Zhou, Sixiang Chen, Chuyan Xiong,
Jiaxin Ge, Renrui Zhang, and Shanghang Zhang. Self-corrected multimodal large language model
for end-to-end robot manipulation. arXiv preprint arXiv:2405.17418, 2024b.

Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu.
Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning.
arXiv preprint arXiv:2105.04165, 2021.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yue Li, Hannaneh Hajishirzi, Hao Cheng,
Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual
contexts with gpt-4v, bard, and other large multimodal models. ArXiv, abs/2310.02255, 2023.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David
Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work:
Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114,
2021.

OpenAI. Chatgpt. https://chat.openai.com, 2023a.

OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023b.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

OpenAI. GPT-4V(ision) system card, 2023c. URL https://openai.com/research/

gpt-4v-system-card.

Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue,
Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model
achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision. In International Conference
on Machine Learning, 2021. URL https://api.semanticscholar.org/CorpusID:
231591445.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL
https://arxiv.org/abs/2305.18290.

Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy
Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language
models. arXiv preprint arXiv:2406.17294, 2024.

Chang Shu, Baian Chen, Fangyu Liu, Zihao Fu, Ehsan Shareghi, and Nigel Collier. Visual med-alpaca:

A parameter-efficient biomedical llm with visual capabilities, 2023.

Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen
Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering
with large language models. arXiv preprint arXiv:2305.09617, 2023.

Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to

instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023.

InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities,

2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation
and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.

Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring
multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804,
2024a.

Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song,
Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in LLMs for enhanced
mathematical reasoning. In The Twelfth International Conference on Learning Representations,
2024b. URL https://openreview.net/forum?id=z8TW0ttBPp.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in
Neural Information Processing Systems, 35:24824–24837, 2022.

Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm:
Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911,
2023.

Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao,
Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models
for 3d lidar understanding. arXiv preprint arXiv:2312.14074, 2023.

14

Under review as a conference paper at ICLR 2025

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen
Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng
Tian, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language
models with multimodality, 2023a.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei
Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with
modality collaboration, 2023b.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo
Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for
large language models. arXiv preprint arXiv:2309.12284, 2023.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu
Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin,
Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert
agi. arXiv preprint arXiv:2311.16502, 2023a.

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint
arXiv:2309.05653, 2023b.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the

web. arXiv preprint arXiv:2405.03548, 2024.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao,
and Yu Qiao. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized
attention. In The Twelfth International Conference on Learning Representations, 2024a. URL
https://openreview.net/forum?id=d4UiXAHN2W.

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou,
Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the
diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024b.

Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia,
Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code
interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921, 2023.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A APPENDIX

A.1 RELATED WORK

Visual Instruction Tuning. The advancement of large language models (LLMs) (Brown et al., 2020;
Jiang et al., 2024; Touvron et al., 2023b; Chiang et al., 2023) with instruction tuning has significantly
enhanced zero-shot capabilities across a range of tasks. Drawing inspiration from this, LLaMA-
Adapter series (Zhang et al., 2024a; Gao et al., 2023b; Han et al., 2023) propose a zero-initialized
attention mechanism to align frozen vision encoders (Radford et al., 2021) with LLaMA (Touvron
et al., 2023a) for multi-modal learning. LLaVA series (Liu et al., 2023b;a) employ a linear projector
for vision-language alignment, establishing visual instruction tuning as a standard training approach
in the multi-modal field. Flamingo (Alayrac et al., 2022) and OpenFlamingo (Awadalla et al., 2023)
have honed visual representation by integrating a cross-attention resampler with vision encoders.
SPHINX series (Gao et al., 2024; Lin et al., 2023) utilize a blend of visual encoders to make the LLM
cognizant of various image aspects. InternVL series (Chen et al., 2024; Dong et al., 2024; Team,
2023) employ a large vision encoder and QFormer (Li et al., 2022) to incorporate high-quality visual
information through a multi-stage training methodology. LLaVA-NexT (Liu et al., 2024a; Li et al.,
2024a;b) further introduces the ‘AnyRes’ technique to manage images at any given resolution, and
LLaVA-NexT-Interleave (Li et al., 2024c) extends the scope widely to interleave multi-image settings.
There are also recent efforts to apply visual instruction tuning to 3D (Guo et al., 2023; Xu et al.,
2023) and video (Li et al., 2023a; Fu et al., 2024) scenarios. Despite the impressive strides made
in both model capability and training efficiency by multi-modal large language models (MLLMs)
through visual instruction tuning, there is currently no MLLM specifically designed for mathematical
problem-solving, nor a substantial dataset available for such purposes in the open-source community.
In this paper, we mitigate the issue by proposing MAVIS with high-quality mathematical visual
datasets and training paradigms.

Mathematics in Large Models. Recent research has predominantly concentrated on text-only
mathematical problem-solving using LLMs. MAmmoTH (Yue et al., 2023b; 2024) have compiled ex-
tensive collections of mathematical problems, training LLMs using the reasoning processes described
in solutions. MetaMATH (Yu et al., 2023) has expanded upon this by rewriting existing problems
to create a larger dataset. MathCoder (Wang et al., 2024b) and ToRA (Gou et al., 2023) introduced
a tools agent approach, employing Python code and symbolic resolvers during the training phase,
significantly outperforming traditional models that rely on text-only mathematical reasoning. How-
ever, in the multi-modal field, despite the introduction of several datasets such as Geometry3K (Lu
et al., 2021), GeoQA (Chen et al., 2021b), UniGeo (Chen et al., 2022), UniMath (Liang et al., 2023),
and GeomVerse (Kazemi et al., 2023), aiming at enhancing the performance of MLLMs in solving
graphical mathematical problems, these datasets are quite limited in scale and domain. Based on
these datasets, G-LLaVA (Gao et al., 2023a) has developed superior capabilities for understanding
graphical geometries but struggles with mathematical problems in other domains. The comprehensive
benchmark MathVerse (Zhang et al., 2024b) has also highlighted the existing MLLMs’ unsatisfactory
capacity for encoding visual diagrams in diverse mathematical domains. Therefore, there is a pressing
need for the development of more robust encoders for mathematical images and the tuning of MLLMs
with mathematical visual instructions, for which we propose MAVIS to address the challenges.

A.2 HUMAN EVALUATION OF MAVIS-INSTRUCT

To assess the dataset’s coverage, validity, and quality, human verification is employed. The creation
process of our MAVIS-Instruct dataset can be broadly categorized into two approaches:

• GPT-generated: This method leverages GPT-4 to generate new problems (including ques-
tions, rationales, and answers) based on existing problems with diagrams. While this
approach produces fluent, human-like sentences, it may be influenced by the inherent
capabilities and occasional instability of GPT-4V.

• Data Engine: As the main source of our mathematical visual data, this method utilizes the
custom automatic data engine to generate new problems (including diagrams, questions,
rationales, and answers), without relying on GPT models. It guarantees 100% correctness
due to the use of rigorous templates, though it may occasionally exhibit rigid expressions.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Figure 7: Human Evaluation Results on
200 randomly sampled problems in MAVIS-
Instruct, 100 GPT-generated and 100 Data
Engine. We set three levels (1, 2, and 3) for
each metric, and report average scores.

Figure 8: Human Evaluation Statistics on
200 randomly sampled problems in MAVIS-
Instruct, 100 GPT-generated and 100 Data
Engine. We count the numbers of three score
levels (1, 2, and 3) for each metric.

Specifically, we evaluate four aspects(Diagram, Question, Rationale and Answer) of each problem
using seven metrics. Each metric is scored on a scale of 1 to 3, where 1 denotes poor, 2 denotes
moderate, and 3 denotes good. The human evaluation results are shown in Figure 7 and score
statistics are shown in Figure 8. In addition, we also showcase some specific examples in Figure 9
and Figure 10. We analyze each aspect as follows:

• Diagram: The diagrams in GPT-generated problems are directly collected from existing
sources with rigorous human filtering, ensuring high quality, resulting in scores close to 3.
In contrast, for rule-based problems, the diagrams are drawn accurately using Python code
driven by our data engine, which guarantees correctness. However, these diagrams may
lack alignment with human aesthetic preferences, as indicated by 3% of them receiving an
appearance score of 1.

• Question: Regarding the questions, both GPT-generated and rule-based problems display
a high degree of accuracy in aligning with the diagram elements. This is attributed to the
well-crafted prompts used with GPT-4 and the meticulous template design of the data engine.
Nevertheless, rule-based questions may occasionally exhibit minor fluency issues, as they
lack human refinement.

• Rationale: In terms of the rationales, most instances feature a precise and detailed chain-of-
thought (CoT) reasoning process. However, in a few cases (3% receiving an accuracy score
of 1), some GPT-generated rationales contain minor reasoning or calculation errors, which
are inherent to GPT-4’s limitations in problem-solving. These errors usually affect only one
or two steps and do not compromise the overall logic. Conversely, the rule-based rationales
are highly accurate due to the carefully designed data engine, although there is still room for
improvement in language fluency.

• Answer: The answers in both methods achieve high correctness scores. For GPT-generated
problems, we prompt GPT-4 to identify a known condition from the original problems as
the answer. Similarly, for rule-based problems, we randomly select a known attribute from
the generated diagrams to serve as the answer.

Overall, the randomly sampled instances show that our dataset exhibits good question quality and
answer accuracy.

A.3 ABLATION STUDY

A.3.1 MAVIS-CAPTION

To validate the enhancement of Math-CLIP’s diagram perception capability, we sampled 100
validation diagram-caption pairs and computed their cosine similarity using both CLIP and Math-
CLIP. The results, as shown in Table 5, indicate that Math-CLIP encodes more discriminative diagram

17

DiagramQuestionRationaleAnswer2.963.002.832.973.002.853.002.913.003.00GPT-generatedData Engine3.002.923.003.003.002.00100cases961001001008810097861009210091100100431138939DiagramQuestionRationaleAnswerScore 3Score 2Score 1GPT-Rule-GPT-Rule-GPT-Rule-GPT-Rule-GPT-Rule-GPT-Rule-GPT-Rule-DiagramQuestionRationaleAnswer2.963.002.832.973.002.853.002.913.003.00GPT-generatedRule-based3.002.923.003.003.002.00100cases961001001008810097861009210091100100431138939DiagramQuestionRationaleAnswerScore 3Score 2Score 1GPT  EngineGPT  EngineGPT  EngineGPT  EngineGPT  EngineGPT  EngineGPT  EngineUnder review as a conference paper at ICLR 2025

Figure 9: Diagram Examples in MAVIS-Instruct. The first three diagrams showcase superior
correctness and appearance, while a small portion of Data Engine generated diagrams (3%) are not
aligned with human preference, e.g., the fourth diagram.

Figure 10: Accurate Rationale Examples in MAVIS-Instruct. Most GPT-generated and Data
Engine-generated rationales ensure correctness.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

GPT-generatedAppearance Score: 3Date EngineAppearance Score: 3Date EngineAppearance Score: 3Date EngineAppearance Score: 1Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Table 3: Statistics of MAVIS-Caption.

Table 4: Subject Distribution of MAVIS-Instruct.

Statistic

Number

Statistic

Total Captions
- Total number
- Average length (words)
- Average length (characters)
- Vocabulary size

Plane Geometry
- Total number
- Average length (words)
- Average length (characters)
- Vocabulary size

Analytic Geometry
- Total number
- Average length (words)
- Average length (characters)
- Vocabulary size

Function
- Total number
- Average length (words)
- Average length (characters)
- Vocabulary size

588K
62.85
339.68
418

299K (50.9%)
69.77
385.85
195

77K (13.1%)
39.64
210.10
158

212K (36.0%)
61.48
321.46
149

Total questions
- Multiple-choice questions
- Free-form questions

Data Engine Generated Problems
- Geometry questions
- Function questions

Data Engine Captions Annotated by GPT-4
- Geometry questions
- Function questions

Manual Collection Augmented by GPT-4
- Geometry questions
- Function questions

Existing Datasets Augmented by GPT-4
- Geometry questions
- Function questions

Number of unique images
Number of unique questions
Number of unique answers

Average question length
Average answer length

Number

834K
615K (62.4%)
218K (37.6%)

582K
466K (80.0%)
116K (20.0%)

51K
30K (58.8%)
21K (41.2%)

83K
72K (86.5%)
11K (13.5%)

118K
118K (100.0%)
0 (0%)

611K (73.3%)
804K (96.5%)
675K (81.0%)

44.60
62.82

features. Additionally, the attention visualization in Figure 1(a) of the main paper further demonstrates
that Math-CLIP captures mathematical visual elements within diagrams more effectively, highlighting
the efficacy of MAVIS-Caption.

To validate the role of MAVIS-Caption in second-stage training, we present both quantitative and
qualitative results for diagram captioning on the same 100 validation pairs in the first column of
Table 6. The use of MAVIS-Caption significantly enhances the diagram understanding capability.
This shows that MAVIS-Caption helps the LLM generate accurate captions from diagrams, improving
its ability to comprehend each visual token from Math-CLIP and align visual elements with textual
descriptions. We also evaluated MAVIS’s performance on MathVerse without second-stage training,
as shown in the second column of Table 6. Without MAVIS-Caption training, the CoT reasoning
quality of MAVIS-7B is somewhat compromised. This suggests that training the model in diagram
captioning improves its mathematical expression capability, enabling it to produce language
expressions that align with mathematical concepts. This foundational skill supports the generation of
subsequent CoT reasoning steps.

Table 5: Diagram Perception Enhancement by Math-CLIP, using MAVIS-Caption in the first
stage. We calculate the average cosine similarity among 100 validation diagram-caption pairs.

Vision Encoder Matched Pair ↑ Unmatched Pair ↓

CLIP
Math-CLIP

0.22
0.83

0.24
0.17

Table 6: Diagram Understanding Enhancement and Mathematical Expression Enhancement in
LLM using MAVIS-Caption in the second Stage. We compare the METEOR and CIDEr scores for
diagram captioning on 100 validation samples, as well as the accuracy and CoT evaluation results on
MathVerse, both with and without the MAVIS-Caption training.

Training Data

Diagram-Caption Pairs

MathVerse

METEOR

CIDEr

Acc (%) CoT-E (%)

w MAVIS-Caption
w/o MAVIS-Caption

23.7
14.0

161.3
69.4

28.4
25.6

35.2
32.8

19

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

A.3.2 MAVIS-INSTRUCT

Redundant Text When curating questions for MAVIS-Instruct, we minimize the redundant content
within the question texts, which refers to the directly observable content in the diagram, e.g., the
presence of shapes or intersection points of functions. Such information is repetitive to visual
components, and may assist MLLMs in bypassing the process of diagram interpretation, thereby
harming their related skills. By mostly avoiding redundant texts in MAVIS-Instruct, our data enforces
MLLMs to learn stronger diagram interpretation capabilities. In Table 7, we add redundant
texts (diagram captions) to the Data Engine Generated Problems for training, leading to expected
performance drop.

CoT Rationales
For each instance in MAVIS-Instruct, we incorporate detailed rationales for
problem-solving, either generated by GPT-4 or our rule-based data engine. In Table 8, we remove all
intermediate rationales of each problem in MAVIS-Instruct, and train the model to directly output
the final answer. As shown, both the CoT evaluation and accuracy scores are degraded. This
demonstrates the significance of our rationale annotations, which effectively improves the CoT
reasoning capabilities of MLLMs.

Table 7: Diagram Interpretation Enhancement for MLLM,
using MAVIS-Instruct in the third stage. We compare the
results by adding redundant texts (diagram captions) to the
Data Engine Generated Problems within MAVIS-Instruct.

MAVIS-Instruct

MathVerse GeoQA FunctionQA

w/o Redundant Texts
w Redundant Texts

28.4
26.5

68.3
66.5

50.0
48.4

Table 8: Reasoning Capability
Enhancement for MLLM, using
MAVIS-Instruct in the third stage.

Training Data

MathVerse

Acc CoT-E

w Rationales
w/o Rationales

28.4
25.2

35.2
26.6

A.3.3 COMPARED TO GENERAL VISUAL INSTRUCTION DATA

Since Mammoth-2 is a highly capable LLM for mathematical tasks, one possible question is whether
simply integrating a vision encoder into Mammoth-2 and training it with conventional visual in-
struction tuning data would suffice for effectively solving visual-based mathematical problems. To
compare MAVIS data with other visual instruction tuning datasets and investigate the specific ben-
efits of MAVIS data in Mammoth-2 (7B), we conduct an ablation study. We utilize the data from
LLaVA-NeXT (558K for pre-training and 760K for fine-tuning) and compare it with our MAVIS data
(558K MAVIS-Caption for pre-training and 834K MAVIS-Instruct for fine-tuning). Performance is
evaluated using the accuracy metric on MathVerse, excluding the DPO training stage for fairness.

Table 9: Ablation study results for comparison between MAVIS Data and other visual instruction
tuning data. The first row in the table represents the original LLaVA-NeXT-8B.

Visual Encoder LLM

Pre-training

Fine-tuning

MathVerse Acc (%)

CLIP
CLIP
CLIP
CLIP
Math-CLIP

LLaVA data
LLaMA-3 (8B)
LLaVA data
LLaVA data
Mammoth-2 (7B) LLaVA data
MAVIS-Instruct
Mammoth-2 (7B) LLaVA data
Mammoth-2 (7B) MAVIS-Caption MAVIS-Instruct
Mammoth-2 (7B) MAVIS-Caption MAVIS-Instruct

15.6
18.3
25.7
26.4
27.5

Based on the results presented in Table 9, we make the following observations:

1. Mammoth-2 vs. LLaMA-3: Mammoth-2 achieves a +2.7 improvement in accuracy com-
pared to LLaMA-3, highlighting its prior knowledge and inherent capability in mathematical
problem solving.

2. Impact of MAVIS-Instruct: Fine-tuning with MAVIS-Instruct significantly enhances
performance by +7.4, underscoring the substantial advantage of our dataset for mathematical
reasoning tasks compared to general visual instruction datasets.

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

3. MAVIS-Caption and Math-CLIP: Using MAVIS-Caption for pre-training and employing
the Math-CLIP encoder further boosts performance, leading to enhanced mathematical
visual perception and reasoning capabilities. Overall, our MAVIS data contributes a +9.2
improvement in accuracy over Mammoth-2 trained with LLaVA data.

A.3.4 PERFORMANCE ACROSS DIFFERENT SUBJECTS

Although MAVIS-Instruct contains a substantial number of high-quality solid geometry problems
that were manually curated, our data engine only generates plane geometry and function problems.
Therefore, we aim to evaluate the performance of the MAVIS model across different mathematical
domains, specifically plane geometry, functions, and solid geometry. We provide the detailed subject
scores of MAVIS-7B on MathVerse, comparing the CoT evaluation score (note that the subject-level
accuracy scores are not publicly released) with other models on the official leaderboard.

Table 10: Performance comparison across different models on Plane Geometry, Solid Geometry, and
Functions of MathVerse evaluation tasks.

Model

All (CoT-Eval) Plane Geometry

Solid Geometry Functions

LLaVA-NeXT
ShareGPT4V
SPHINX-MoE
InternLM-XC2
MAVIS-7B

17.2
17.4
22.8
25.9
35.2

15.9
16.9
24.5
26.2
37.1

19.6
15.0
15.8
20.1
28.9

23.1
20.2
19.5
23.7
31.0

The results shown in Table 10 demonstrate that our model achieves leading performance across all
three subjects. Notably, its proficiency in plane geometry and functions can be attributed to the
training with our meticulously curated MAVIS dataset. Additionally, for solid geometry, which shares
similarities with plane geometry in both visual appearance and reasoning process, we believe that our
model effectively generalizes its learned knowledge and reasoning capabilities, leading to enhanced
performance in this domain as well.

A.3.5 SYNTHETIC DATA VS REAL DATA

In MAVIS-Instruct, we integrate both synthetic problems generated by the data engine (633K, 76%)
and real-world problems augmented with GPT (201K, 24%). The synthetic data is composed of
both geometry and function problems, while the real-world data primarily focuses on geometry. We
conduct an ablation study to assess the contributions of these data components, excluding the DPO
training stage to ensure fairness.

Table 11: Ablation study of synthetic and real data contributions to MAVIS-7B’s performance.

Synthetic Data Real-world Data MathVerse Acc (%) GeoQA FunctionQA MMMU-Math

✓
–
✓

–
✓
✓

22.6
24.3
27.5

44.2
66.4
66.7

37.1
25.8
40.3

34.6
29.8
39.2

The results shown in Table 11 indicate that the two data sources exhibit complementary characteristics,
both playing a crucial role in achieving the final performance. Specifically, synthetic data significantly
enhances the results on FunctionQA and MMMU-Math, as these benchmarks include a substantial
proportion of function-related problems. Conversely, real-world data has a greater impact on GeoQA,
given its stronger alignment with the geometry-focused nature of this benchmark.

A.3.6 DATA SCALING

A good instruction tuning dataset should exhibit the characteristic of data scaling: as the dataset size
increases, the model trained on it should demonstrate progressively better performance. To verify that
MAVIS-Instruct possesses this property, we conduct an ablation study on the 834K MAVIS-Instruct
dataset by randomly sampling 25%, 50%, and 75% of the data for instruction tuning, excluding

21

Under review as a conference paper at ICLR 2025

the DPO stage. We then evaluate the models using the accuracy metric on MathVerse. The results,
as shown in Table 12, indicate that the performance of MAVIS-7B consistently improves as the
data scale increases. This demonstrates the promising potential of our dataset to further enhance
mathematical reasoning capabilities with larger-scale utilization.

Table 12: Performance of MAVIS-7B at different
data proportions.

Table 13: Comparison of different training set-
tings.

25% 50% 75% 100%

23.3

25.7

26.9

27.5

LLMs

Caption CIDEr MathVerse Acc (%)

Frozen
Unfrozen
LoRA-based

79.6
146.2
161.3

26.2
28.1
28.4

A.3.7 GENERALIZATION ABILITY

Although our Data Engine considers as many problem types as possible, it is inherently challenging
for a manually designed engine to cover all possible types of mathematical visual diagrams. To
alleviate this, we incorporate real-world data by manual collection and GPT augmentation, which
well enhances the generalization capablitity of MAVIS-7B. In Figure 11, we selected examples
involving inscribed triangles (they cannot be generated via our data engine) from publicly available
math benchmarks, and tested MAVIS-7B on these examples, where our model performs accurate
reasoning.

Figure 11: Examples for MAVIS-7B solving math problems with circumcircles of triangles.

A.3.8 ENHANCING VISION-ONLY PERFORMANCE

To evaluate the impact of OCR datasets on MAVIS-7B’s performance, we conducted an ablation
study focusing on vision-only and vision-dominant problems in MathVerse. These problems require
the model to interpret question texts rendered directly in diagrams, thus relying heavily on OCR
capabilities. MAVIS-7B, however, was initially not trained with OCR-specific datasets, limiting its
performance in these tasks.

In contrast, generalist models like LLaVA-NeXT include extensive OCR datasets such as OCRVQA,
DocVQA, and SynDog-EN, which significantly enhance their OCR capabilities. To bridge this gap,
we incorporated OCR datasets (OCRVQA and DocVQA) in our third-stage instruction tuning to
improve MAVIS-7B’s OCR performance.

The results, as shown in Table 14, indicate a notable improvement in vision-dominant and vision-only
problems for MAVIS-7B after the inclusion of OCR datasets, highlighting the potential of better OCR
integration for further boosting its performance. In Figure 12, we also showcase some failure cases
of our MAVIS-7B with OCR training on vision-only problems. Although the vision-only results are
improved via the OCR instruction dataset, the model still suffers from limited perception capabilities
of questions and visual elements within the diagram. This indicates that the OCR capability is still
the bottleneck of vision-only performance. We leave this as a future work to further enhance the
OCR capabilities of MAVIS for mathematical visual elements.

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Q:ABistangenttocircleOatpointB,andtheextendedlineofAOintersectscircleOatpointC.ConnectBC,ifangleA=36.0,thenangleCisequalto?A:SinceABistangenttocircleOatpointB,wehaveOBperpendiculartoAB.Therefore,angleABOis90°.AngleAOBisequalto90°-angleA,whichis90°-36°,givingus54°.SinceOCisequaltoOB,angleCisequaltoangleOBC.AndsinceangleAOBisequaltoangleCplusangleOBC,wehaveangleCequaltohalfofangleAOB,whichishalfof54°,givingus 27°.Q:incircleO,chordACparallelradiusOB,angleBOC=50.0,thedegreeofangleOABis?A:SinceACisparalleltoOB,thereforeangleBOCisequaltoangleACOwhichis50degrees.Also,OAisequaltoOC,soangleOACisequaltoangleACOwhichis50degrees.SinceangleCABishalfofangleBOC,itis25degrees.Therefore,angleBAOisequaltoangleOACminusangleCABwhichis25degrees.Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Figure 12: Failure cases of MAVIS-7B with OCR training on vision-only problems.

Table 14: Impact of OCR data on solving vision-only problems.

Model

LLM Size All

Text Dominant Text Lite Vision Intensive Vision Dominant Vision Only

LLaVA-NeXT
LLaVA-NeXT
MAVIS-7B
MAVIS-7B w/ OCR

8B
110B
7B
7B

15.6
24.5
28.4
28.9

19.4
31.7
41.6
40.8

15.2
24.1
29.5
29.2

16.8
21.0
27.9
27.4

15.2
22.1
24.7
26.2

11.3
20.7
18.3
21.1

A.3.9 BASE LLM

We investigate different LLMs for the MAVIS model. As shown in Table 15, MAVIS is not very
sensitive to LLM choices, and still surpasses previous models with the same LLM.

Table 15: Performance Comparison using
Different LLMs. We compare the accuracy
and CoT evaluation results on MathVerse.

Method

Base LLM

MathVerse

Acc CoT-E

SPHINX-Plus
ShareGPT4V
InternLM-XC2.

LLaMA2-13B 12.2
13.1
Vicuna-13B
InternLM2-7B 16.5

MAVIS

LLaMA2-13B 24.5
Vicuna-13B
24.8
InternLM2-7B 28.0
28.4
MAmmoTH2

14.0
17.4
25.9

30.7
30.6
33.8
35.2

A.3.10 DIFFERENT TRAINING SETTINGS

Our training strategy is similar to LLaVA, but with key differences in the pre-training stage: we
train both the projection layer and the LoRA-based LLM, whereas LLaVA only trains the projection
layer. This design choice stems from the fundamental differences between general visual tasks and
mathematical tasks:

1. For general visual tasks (e.g., LLaVA), training MLLMs typically requires the LLM to generate
daily natural language responses, such as descriptive captions or instruction-following outputs.
These outputs often rely on pre-existing knowledge within the pre-trained LLM. As a result, in
LLaVA, there is no need to unfreeze the LLM to learn new types of outputs.

23

Since BD bisects angle ABC, angles ABD and CBD are congruent. With angle ABC measuring 70 degrees (180 - 70 - 40 = 70), each of these angles measures 35 degrees (70 / 2 = 35). Because CD is parallel to AB, line BD acts as a transversal, … so angle CDB also measures 35 degrees. Therefore, the answer is D. In the given diagram, AB is tangent to the circle at point B, and ∠OBA=90∘ since a tangent is perpendicular to the radius at the point of tangency. The triangle △OBC is isosceles as OB=OC (both are radii of the circle). Given ∠ABO=36∘, the remaining angle ∠OBC=180∘−90∘−36∘=54∘. Since △OBC is isosceles, ∠C=∠OBC=54∘. Thus, the correct answer is B: 54°.To solve 10×f(7)+9×g(1), we first determine f(7)and g(1) from the graphs. From the blue graph (y=f(x)), at x=7, f(7)=−6. From the pink graph (y=g(x)), at x=1, g(1)=2. Substituting these values into the expression gives 10×(−6)+9×2=−60+18=−42. Thus, the final answer is -42.Question	&	Diagram:Answer:AAnswer:DAnswer:	−1Question	&	Diagram:Question	&	Diagram:MAIS-7B with OCR:MAIS-7B with OCR:MAIS-7B with OCR:Under review as a conference paper at ICLR 2025

2. In contrast, for mathematical domains, LLMs need to generate math-specific responses, such as
geometric descriptions, functional explanations, formulas, and theorems. These outputs often
involve domain-specific knowledge not inherent in pre-trained LLMs. Given this, we incorporate
learnable LoRA layers to infuse new knowledge into the LLM, enhancing its capability to produce
high-quality mathematical expressions. Concurrently, we aim to prevent the LLM from overfitting
to diagram captioning tasks during alignment. Therefore, using LoRA-based tuning allows us to
preserve the LLM’s generalizable pre-trained language knowledge while injecting specialized
math-specific capabilities.

To further investigate the impact of different training settings on model performance, we conduct
an ablation study comparing various LLM training settings during the alignment stage. We evaluate
two tasks: the CIDEr score for diagram captioning on 100 validation samples (following the same
setting as in Table 6 of the Appendix) and the accuracy score on MathVerse. The results, as shown in
Table 13, indicate that the LoRA-based approach performs best, enabling MLLMs to generate high-
quality mathematical captions while preserving pre-trained knowledge for improved problem-solving
capabilities.

A.3.11 ENHANCING A PRE-TRAINED MLLM

To investigate whether our curated data and training techniques can improve the mathematical perfor-
mance of a pre-trained large model (LLaVA-NeXT), we conducted an ablation study. Specifically,
we progressively employed MAVIS-Instruct for instruction tuning, followed by DPO alignment
on top of LLaVA-NeXT-8B, with both training stages performed for one epoch using a learning
rate of 1 × 10−5. The results, as shown in Table 16, demonstrate that these two continual training
stages significantly enhance LLaVA-NeXT’s ability to solve mathematical problems, with notable
improvements across all evaluation categories.

Table 16: Performance improvement of LLaVA-NeXT-8B with MAVIS-Instruct and DPO alignment.

Model

LLM Size All

Text Dominant Text Lite Vision Intensive Vision Dominant Vision Only

LLaVA-NeXT
+ MAVIS-Instruct
+ DPO

8B
8B
8B

15.6
22.8
24.0

19.4
32.3
33.7

15.2
25.3
26.9

16.8
24.6
25.4

15.2
18.3
19.1

11.3
14.2
15.1

A.4 DETAILS OF AUTOMATIC DATA ENGINE

A.4.1 DIAGRAM GENERATION

In this section, we detail the implementation specifics of the process for generating diagrams related
to plane geometry, analytic geometry, and function domains.

Plane Geometry Diagram.
Inspired by previous multi-hop reasoning methods (Kazemi et al., 2023;
Wei et al., 2022; Nye et al., 2021), we employ an iterative generation method over logical theories
to generate plane geometric images along with corresponding captions and question-answering
pairs, whose complexity can be controlled across multiple axes. Specifically, we first define a set of
fundamental geometric shapes in Figure 13.

Figure 13: The set of fundamental shapes in plane geometry diagrams, whose straight edges can be
extended into other basic shapes.

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Within each shape, new basic shapes can be generated by extending a particular edge. For each basic
shape, we initially define a meta reasoning process:

On−1, C i

mn−1

Ei
mn−1−−−−→ On, i ∈ [1, z],

(1)

where O represents the initial side length of the shape, Cm denotes the additional conditions required
to complete meta reasoning, and Em provides a detailed explanation of the meta reasoning process.
For example, when considering an isosceles triangle as the (n − 1)th shape in a sequence, the vertex
angle is still required as Cm to reason about base side length, and then to expand to the nth shape,
with Em serving as the explanation of this process. The variable z indicates that there are z sets of
possible meta reasoning for the shape, n indicates the length of the generating sequence, which is
also the number of hops of reasoning required to answer the question. The initial side, extend side,
and additional conditions for meta-reasoning of each basic shape can be referred to in Figure 13. In
the final shape, question-answering pairs pertinent to this shape can be generated as

On, C j
qn

, Qj
n

Ej
qn−−→ Aj

n, j ∈ [1, m],

(2)

where Cq represents the additional conditions required to solve the problem, while Q and A denote
the question and answer, respectively. Eq refers to the detailed explanation of the solving process.
The variable m indicates that there are m pairs of question-answering and corresponding detailed
explanations within the shape. By applying meta reasoning to the n − 1th shape, the initial side
length of the nth shape can be deduced. Therefore, for a complex composite figure consisting of n
shapes, the overall question-answering pair can be defined as follows:

O1,

n−1
(cid:88)

k=1

Cmk , C j
qn

, Qj
n

Ej
qn−−→ Aj
n.

(3)

Each shape defines a sufficient number of conditions, explanations, and answers to ensure the diversity
of the generated question-answering pairs. Based on the aforementioned rules, controlling the length
of the generation sequence can regulate the number of reasoning steps, and controlling the type of
questions can manage the knowledge required for solving the problems. Thus, we can generate
questions of varying difficulty levels, which can also be illustrated in Figure 14a.

Analytic Geometry Diagram. The image generation method for analytic geometry is relatively
straightforward. First, we randomly select a range within the coordinate system: the minimum value
of x is chosen as an integer between [−12, −8], and the maximum value of x is chosen as an integer
between [8, 12]; the range for y is the same as for x. Then, we define the following basic shapes:
point, line segment, line, circle, ellipse, rectangle, square, polygon, and sector. During the generation
process, we select a number between 1 and 4 as the number of shapes to generate. The generation
rule is that nonlinear shapes other than points, line segments, and lines must not overlap.

Function Diagram. The generation of function graphs is also straightforward as shown in Fig-
ure 14b. We define the following basic functions, each with a set of parameters that can be randomly
selected:

Sine Function

Cosine Function

Tangent Function

Polynomial Function

y = A · sin(f · x + ϕ), where the amplitude A is a random integer
between 1 and 3, the frequency f is either 1 or 2, and the phase ϕ is a
random integer between 0 and 2π.

y = A · cos(f · x + ϕ), where the amplitude A is a random integer
between 1 and 3, the frequency f is either 1 or 2, and the phase ϕ is a
random integer between 0 and 2π.

y = A · tan(f · x + ϕ), where the amplitude A is a random integer
between 1 and 3, the frequency f is either 1 or 2, and the phase ϕ is a
random integer between 0 and 2π.
P (x) = anxn + an−1xn−1 + · · · + a1x + a0, where the degree n is
a random integer between 1 and 4. The coefficients ai are randomly
selected integers ranging from -3 to 3.

25

Under review as a conference paper at ICLR 2025

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

(a) A single process for generating plane geometry diagrams and corresponding question-answering pairs as
well as image captions. In this example, the generation sequence length is specified as 2. Initial side length is
painted in pink, Cm is painted in green, while Cq is painted in yellow. Whenever a new basic shape is generated,
its caption is appended to the previous caption.

(b) A single process is used for generating function diagrams along with the corresponding question-answer pairs
and image captions. Once the functional expression is determined, all its properties can be directly computed,
and the function plot can be generated accordingly. The caption for the function diagram simply states the
functional expression.

Figure 14: The pipeline of our data engine, consisting of (a) the generation of plane geometry
diagrams and (b) the generation of function diagrams.

piece-wise Function

piece-wise polynomial functions are divided into 2 or 3 segments, with
each segment’s parameters identical to those of a polynomial function.

Logarithmic Function

y = a · logb(c · x + d), where the coefficient a is randomly cho-
sen from {−3, −2, −1, 1, 2, 3}, the base b is randomly chosen from
{2, 10, ⌊e⌋}, the coefficient c is a random integer between 1 and 3,
and the coefficient d is a random integer between 1 and 6, ensuring
that c · x + d is positive.

Absolute Function

y = |a · x + b|, where a and b are random integer between −5 and 5.

We first determine the domain range to be displayed on the function graph. For trigonometric
functions, the domain is set to [−π, π]. For piece-wise polynomial functions, the minimum value of
x is a random integer between [−12, −8], and the maximum value of x is a random integer between
[8, 12]. For other functions, the minimum and maximum values of x are random integers within
the ranges of [−6, −3] and [3, 6], respectively. During the plotting process, we calculate the local
maxima, minima, and zeros of the function by iterating through the domain. We then render the
x-coordinates of these extrema and zeros on the x-axis of the function graph.

26

Under review as a conference paper at ICLR 2025

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Figure 15: Examples of analytical geometry diagram caption.

A.4.2 MAVIS-CAPTION

In this section, we detail how the captions corresponding to images in the MAVIS-Caption Dataset
are generated with our automatic data engine.

Plane Geometry Caption. Based on the generation process described in Section A.4.1, when
generating each shape, a caption is randomly selected from a set of captions for that shape and some
connecting words are randomly added. We also randomly select some edges or angles and state their
measurements in the caption. After generating the raw caption, we use GPT-3.5 to refine it, enhancing
its linguistic structure and semantic diversity. An example is shown in Figure ??.

Function Caption. According to the function graph generation process described in Section A.4.1,
we record the function’s zeros and extrema. Additionally, we also record the function’s expression
and asymptotes. These attributes are incorporated into a randomly selected caption template to form
the function graph’s caption. Some examples are provided in Figure 16.

Analytic Geometry Caption. For each shape, we maintain a set of caption templates that describe
the shape’s type, coordinate position, and other attributes. In the generation process described in
Section A.4.1, we select a template and randomly add some diverse connecting words to form a
complete caption. Examples of some captions are shown in Figure 15.

A.4.3 MAVIS-INSTRUCT

Manual Collection Augmented by GPT-4. To complement the dataset with real-world problem-
solving scenarios, we hire 8 human experts to manually collect visual math problems from various
public sources1,2,3, spanning plane geometry, analytic geometry, and function. For problems, we try
to obtain their content as complete as possible, including questions, diagrams, answers, and rationales
if available. The collection process consists of the following steps:

1. Problem Collection: We gathered problems from three public sources as comprehensively as
possible, including questions, diagrams, answers, category information, and rationales where
available. The problems are primarily at the high-school level, covering plane geometry and
functions (including analytic geometry).

2. Data Verification: Based on their initial categories (subject, subfield, and difficulty level), the
problems were organized into distinct groups. Six expert annotators were tasked with meticulously
verifying the correctness and completeness of each problem. They refined the detailed chain-of-
thought (CoT) rationales and ensured that there was no overlap with evaluation data by visually
inspecting the diagrams. This rigorous verification process resulted in a total of 4K verified
problems.

27

Under review as a conference paper at ICLR 2025

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Figure 16: Function diagram captions.

3. Text-lite Construction: To optimize the problems for training mathematical visual capabilities,
the 4K problems were processed using GPT-4V with a customized prompt (as shown in Figure
15). This step involved removing redundant information from the question text to create concise,
text-lite problems, specifically tailored to our training objectives.

Then, we first feed all the related information into GPT-4V to eliminate the redundant information
within text questions, constructing the text-lite version of problems by the prompt in Figure 17. Then,
we design three types of prompts for GPT-4 to augment 15 multiple-choice questions (including 10
multiple-choice and 5 binary-choice, i.e., ‘True’ or ‘False’) and 5 free-form questions, respectively,
as shown in Figure 18. We do not adopt GPT-4V here, since GPT-4V itself would misunderstand
diagrams for low-quality data augmentation. The newly generated problems contain detailed CoT
rationales and diverse question forms.

Figure 17: Manually collect visual math problems text-lite version.

Existing Datasets Augmented by GPT-4. Previous efforts have been made to provide some small-
scale, plane geometry datasets, e.g., GeoQA (Chen et al., 2021c), GeoQA+ (Chen et al., 2021a),
and Geometry3K (Lu et al., 2021). Although they are limited in data scale for tuning MLLMs

1https://homework.study.com
2https://www.ixl.com/math
3https://mathspace.co/us

28

Under review as a conference paper at ICLR 2025

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Figure 18: We design different types of prompts for GPT-4 to augment 15 multiple-choice questions
and 5 free-form questions, respectively.

and include no rationales, we can also regard them as a seed dataset and adopt GPT-4 to augment
larger-scale training data. We do not utilize GPT-4V here for the same reason aforementioned. In
detail, we design 3 types of question generation approaches using different prompts, as shown in
Figure 19. For Geometry3K, as the question texts are normally brief and contain marginal descriptive
information, posing challenges for GPT-4 to understand the diagram, we only augment them to
generate binary-choice questions, i.e., ‘Ture’ or ‘False’. For GeoQA+, we can leverage the sufficient
redundant information within their texts to generate more diverse and accurate multi-choice and
free-form questions. Likewise, GPT-4 can produce CoT rationales for each problem.

Figure 19: We design 3 types of question generation approaches using different prompts to augment
existing visual mathematical dataset.

Data Engine Captions Annotated by GPT-4. Given the delicately designed data engine for
automatic diagram-caption creation, we can utilize the generated large-scale pairs to annotate question-
answering data using GPT-4V. Different from the previous two sources that augment questions based
on questions, we utilize the GPT-4V model here for caution data with two reasons: first, the detailed
caption from our data engine can well guide GPT-4V for relatively higher-quality visual embedding;
second, the visual input serves as guidance to provide additional spatial information for broad

29

Under review as a conference paper at ICLR 2025

Figure 20: The Text Dominant, Text Lite, Vision Dominant, and Vision Only versions of the same
question. Text Dominant and Text Lite use the same image. In the text, the necessary conditions for
solving the problem are highlighted in red, while redundant descriptive conditions are highlighted in
blue. In the Vision Only version, the question is rendered in the image, with no textual format.

question forms. As shown in Figure 27 and Figure 28, we adopt different prompts for function and
plane geometry problems, ensuring that the generated question-answering data is of high quality for
instruction tuning.

Figure 21: Perimeter problem templates.

Figure 22: Area problem templates.

Data Engine Generated Problems:

PLANE GEOMETRY. Based on the generation process described in Section A.4.1, we pose questions
about the final shape in the generation sequence. We designed 6 types of questions: finding the
perimeter, finding the area, finding the base length, finding the angle, finding the arc length, and
finding the extended edge length. Each type of question has a set of templates that can be randomly
selected, as shown in Figure 21-26. As for the answer and analysis, each shape has a set of templates
for different types of questions to choose from, as shown in Section A.4.1.

To further enhance the model’s understanding of different forms of questions and better utilize the
diverse modal information in the text and images, we divided the plain geometry questions generated
by the Data Engine into four versions referring to MathVerse (Zhang et al., 2024b): Text Dominant,
Text Lite, Vision Dominant, and Vision Only.

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Under review as a conference paper at ICLR 2025

Figure 23: Base length problem templates.

Text Dominant

Text Lite

We marked all the conditions required for solving the problem in the diagram
and also described these conditions in the text, along with some redundant
descriptive text.

All the conditions required for solving the problem are randomly divided into
two parts: one part is marked in the diagram, and the other part is described
in the text. In other words, the conditions in the diagram and the conditions
in the text do not overlap.

Vision Dominant

All the conditions required for solving the problem are marked in the diagram,
while the text only contains the question without any conditions.

Vision Only

Not only are all the conditions required for solving the problem marked in
the diagram, but the question is also rendered in the diagram, leaving the text
portion empty.

The differences among the four versions of the same question are illustrated in Figure 20. Each basic
shape will retain a set of redundant conditions. During the shape generation process, there is a 50%
probability of including these redundant conditions.

FUNCTION. All functions will be examined with two types of questions: finding the derivative and
finding the extrema. After obtaining the derivative, we calculate whether the derivative has zeros
within the given domain. The presence of zeros determines the method for calculating the extrema.

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

31

Under review as a conference paper at ICLR 2025

Figure 24: Angle problem templates.

Figure 25: Arc length problem templates.

Figure 26: Extend side length problem tem-
plates.

32

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Under review as a conference paper at ICLR 2025

Figure 27: The function prompt for GPT-4V and the generated questions and rationals.

Figure 28: The geometry prompt for GPT-4V and the generated questions and rationals.

33

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

The function can be described by the equation y = 1*cos(1*x + 3). With an x range of [-𝝿, 𝝿], we find the zero points at -1.43, 1.71. Exploring the function within the range [-𝝿, 𝝿], we find that the highest value it reaches is 1.0 at -3.0, and the lowest value is -1.0 at 0.14. There are no asymptotes in the functions.Question: What is the length of square side FG? Choices: A. 10.0; B. 10.1; C. 9.9; D. 10.5Rational: Since FG is a side of square DBFG, and the properties of a square dictate that all sides are equal in length, the length of FG must be the same as DB which is given as 10.0 units. So the answer is A.Diagram:Caption:Question & Rationale Generated by GPT-4V:Question: What is the length of diagonal AC in triangle ABC? Choices: A. 10.1; B. 9.8; C. 9.9; D. 10.0Rational: In right triangle ABC with sides AB = 7.0 and BC = 7.0 units, and angle ACB = 45°, the triangle is isosceles. The length of diagonal AC can be calculated using the Pythagorean theorem: AC = √(AB² + BC²) = √(7.0² + 7.0²) = √98 ≈ 9.9. So the answer is C.Question: What is the perimeter of quadrilateral CBDE? Choices: A. 33.0; B. 34.0; C. 35.0; D. 32.0Rational: The perimeter of quadrilateral CBDE includes sides CB, BD, DE, and EC. Given CB = 7.0, BD = 10.0, and knowing quadrilateral CBDE has parallel and equal opposite sides, DE = CB and EC = BD. Thus, DE = 7.0 and EC = 10.0. The perimeter = CB + BD + DE + EC = 7.0 + 10.0 + 7.0 + 10.0 = 34.0 units. So the answer is B.Here is some more supplementary information for the diagram I give you: "{caption}"Please first watch the diagram and ask a question that can be correctly answered only by seeing the diagram.Then you should correctly analysis the question and give the answer. The reasoning process should not use "According to the caption" or something like that.Next, you should give 3 other hard false choices along with the correct one.You should totally provide 3 questions.The type of questions is multi-choice. You should strictly follow this format and order (end every problem with ####):QUESTION: What is the maximum value of the function in the given range? ANALYSIS: From the diagram, we can observe that when x=0, the function obtain the maximum value, which is 4.ANSWER: B. 4CHOICE: A. 3; B. 4; C. 2; D. 1 ####Prompt for Function Problems:Firstly, trace a Right Triangle ABC.Secondly, attached to edge CB, draw a Quadrilateral with opposite sides parallel and equal in length CBDE.Thirdly, joined to edge DB, render a Square DBFG.The measurement of angle ACB is 45o. The angle DBC measures 60o. Angle BDG measures 90oin size. The distance between points A and B is 7.0. The segment CB spans 7.0 units. Side DB spans 10.0 units.Question: What is the length of square side FG? Choices: A. 10.0; B. 10.1; C. 9.9; D. 10.5Rational: Since FG is a side of square DBFG, and the properties of a square dictate that all sides are equal in length, the length of FG must be the same as DB which is given as 10.0 units. So the answer is A.Diagram:Caption:Question & Rationale Generated by GPT-4V:Question: What is the length of diagonal AC in triangle ABC? Choices: A. 10.1; B. 9.8; C. 9.9; D. 10.0Rational: In right triangle ABC with sides AB = 7.0 and BC = 7.0 units, and angle ACB = 45°, the triangle is isosceles. The length of diagonal AC can be calculated using the Pythagorean theorem: AC = √(AB² + BC²) = √(7.0² + 7.0²) = √98 ≈ 9.9. So the answer is C.Question: What is the perimeter of quadrilateral CBDE? Choices: A. 33.0; B. 34.0; C. 35.0; D. 32.0Rational: The perimeter of quadrilateral CBDE includes sides CB, BD, DE, and EC. Given CB = 7.0, BD = 10.0, and knowing quadrilateral CBDE has parallel and equal opposite sides, DE = CB and EC = BD. Thus, DE = 7.0 and EC = 10.0. The perimeter = CB + BD + DE + EC = 7.0 + 10.0 + 7.0 + 10.0 = 34.0 units. So the answer is B.Here is some more supplementary information of the diagram I give you: "{caption}"Please first watch the diagram and ask a question that can be correctly answered only by seeing the diagram.Then you should correctly analysis the question and give the answer. The reasoning process should not use "According to the caption" or something like that.Next, you should give 3 other hard false choices along with the correct one.You should totally provide 3 questions.The type of questions is multi-choice. You should strictly follow this format and order (end every problem with ####):QUESTION: What is the height of the trapezium ABCD?ANALYSIS: Since we know the length of AB and the angle CBA, we can derive the height of the trapezium ABCD. The height should be AB \times sin(\angle CBA) = 11.7 * sin(60) = 10.1, so the answer is 11.7ANSWER: B. 11.7CHOICE: A. 11; B. 11.7; C. 12; D. 8 ####Prompt for Plane Geometry Problems:Under review as a conference paper at ICLR 2025

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

Figure 29: The initial side, extend side, and additional conditions for meta-reasoning of each basic
shape. Some special shapes are not extended and only appear in the last position of the generation
sequence, thus their extend side is ∅.

34

