MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code
Under review as a conference paper at ICLR 2025

MATHCODER2: BETTER MATH REASONING FROM
CONTINUED PRETRAINING ON MODEL-TRANSLATED
MATHEMATICAL CODE

Anonymous authors
Paper under double-blind review

ABSTRACT

Code has been shown to be effective in enhancing the mathematical reasoning
abilities of large language models due to its precision and accuracy. Previous
works involving continued mathematical pretraining often include code that uti-
lizes math-related packages, which are primarily designed for fields such as engi-
neering, machine learning, signal processing, or module testing, rather than being
directly focused on mathematical reasoning. In this paper, we introduce a novel
method for generating mathematical code accompanied with corresponding rea-
soning steps for continued pretraining. Our approach begins with the construc-
tion of a high-quality mathematical continued pretraining dataset by incorporat-
ing math-related web data, code using mathematical packages, math textbooks,
and synthetic data. Next, we construct reasoning steps by extracting LaTeX ex-
pressions, the conditions needed for the expressions, and the results of the ex-
pressions from the previously collected dataset. Based on this extracted infor-
mation, we generate corresponding code to accurately capture the mathematical
reasoning process. Appending the generated code to each reasoning step results
in data consisting of paired natural language reasoning steps and their correspond-
ing code. Combining this data with the original dataset results in a 19.2B-token
high-performing mathematical pretraining corpus, which we name MathCode-
Pile. Training several popular base models with this corpus significantly improves
their mathematical abilities, leading to the creation of the MathCoder2 family of
models. All of our data processing and training code is open-sourced, ensuring
full transparency and easy reproducibility of the entire data collection and train-
ing pipeline.

1

INTRODUCTION

Various studies (Azerbayev et al., 2024; Shao et al., 2024) have shown that training on code en-
hances the mathematical reasoning abilities of large language models (LLMs). Previous research in
continued mathematical pretraining often includes code that utilizes math-related packages (Azer-
bayev et al., 2024). This code is typically sourced from GitHub and is primarily designed for fields
such as engineering, machine learning, signal processing, or module testing, rather than focusing
directly on mathematics. Recent models (Zhou et al., 2024; Yang et al., 2024b; Ying et al., 2024;
Shao et al., 2024; Wang et al., 2023a) have adopted Tool-Integrated Reasoning (TIR) in fine-tuning.
They utilize integrated natural language reasoning steps and Python code to improve performance
on mathematical reasoning tasks. Reasoning with the help of code is particularly effective for more
challenging problems, likely due to its precision and accuracy.

Although utilizing existing open-source code in the pretraining phase can enhance the mathematical
reasoning abilities of LLMs, such code often lacks accompanying natural language explanations or
context. This might hinder the model’s ability to fully understand them. In this paper, we propose
a novel method for generating large amounts of mathematical code accompanied by corresponding
natural language reasoning steps, which are extracted from math-related pretraining texts. Different
from the existing math-related code, our generated code is paired with natural language reasoning
steps, making the code more comprehensible. Also, as our code is generated based on math-related
texts, they are all highly related to mathematical reasoning. When used in pretraining, the mathe-

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

matical code paired with reasoning steps facilitates LLMs’ understanding of math-related pretraining
texts, as it effectively captures the underlying reasoning process. Furthermore, this data enhances
the model’s potential to be finetuned for TIR reasoning.

Our data processing pipeline consists of two key steps: (1) carefully curating a robust basic dataset
for pretraining, and (2) generating paired reasoning steps and mathematical code by extracting La-
TeX expressions and their context, translating the extracted information into Python code snippets,
executing the generated code snippets, and verifying their correctness.

First, we gather and carefully filter a wide variety of math-related data sources, including web pages,
model-generated data, math-related code, and textbooks. Through an advanced filtering process, we
ensure the dataset is both large and highly relevant, minimizing irrelevant content while preserving
the mathematical texts necessary for training. This results in a 16.5B-token dataset that forms the
foundation of our pretraining efforts. By conducting experiments with smaller models, we show that
this careful curation leads to more efficient training without sacrificing model performance.

Second, we propose a novel method for generating large amounts of paired mathematical reason-
ing steps and their corresponding Python code. Given a piece of text from the pretraining corpus
collected above, we wrap it in a carefully designed prompt that instructs a Llama-3.1-70B-Instruct
model to extract LaTeX expressions along with their relevant context, including the conditions for
each expression and the result of its computation. This results in a list of comprehensive mathemati-
cal reasoning steps, complete with the necessary conditions, the computations taken, and the results.
Then, we prompt the model to translate each reasoning step into a Python code snippet that captures
the underlying reasoning process. The generated Python snippets are executed, and only those that
run successfully and produce outputs matching the expected results are retained. By pairing the
code with the corresponding reasoning step, we create the final data. The process is demonstrated
in the lower half of Fig. 1. This process yields a 2.7B-token corpus of mathematical code snippets
accompanied with their corresponding reasoning steps, which we combine with the data generated
in the first step to create a 19.2B-token pretraining dataset, named MathCode-Pile.

We validate the effectiveness of MathCode-Pile on four popular base models: Llama-3-8B,
DeepSeekMath-7B, Mistral-7B, and Code-Llama-7B, significantly improving their performance on
five representative mathematical benchmarks. We name the resulting family of pretrained mod-
els MathCoder2. In particular, MathCoder2-Llama-3-8B achieves 4-shot accuracies of 38.4% on
MATH and 69.9% on GSM8K, outperforming the baseline of training only on the basic data gener-
ated in the first step by 3.1% and 4.1%, respectively. This demonstrates that the data of mathematical
code accompanied with reasoning steps effectively enhances LLMs’ reasoning abilities.

Different from recent works, such as DeepSeekMath (Shao et al., 2024), InternLM-Math (Ying
et al., 2024), and Qwen2.5-Math (Yang et al., 2024b), which only release their model weights, we
offer a detailed, open-source framework for data processing and training that achieves performance
competitive with these models, fostering further progress in mathematical reasoning for LLMs.

Our contributions include:

• A novel and effective method for generating large amounts of mathematical code with cor-
responding natural language reasoning steps, significantly enhancing pretraining outcomes.

• The creation of MathCode-Pile, a meticulously curated 19.2B-token dataset for contin-
ued mathematical pretraining. This dataset includes math-related web data, synthetic data,
code, textbooks, and model-translated mathematical code.

• Full open-sourcing of all data processing and training code, ensuring transparency and

reproducibility to support future research.

2 CURATION OF MATHCODE-PILE

We curate our mathematical pretraining dataset, MathCode-Pile, in two steps: first, we collect the
basic data in Sec. 2.1, and then we use it to generate mathematical code snippets with their corre-
sponding natural language reasoning steps in Sec. 2.2.

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Figure 1: The data processing pipeline. (a) shows the pipeline of prior works. (b) demonstrates our
method. We first use a fastText classifier to filter the Common Crawl corpus, resulting in the initial
filtered math texts. Then, we annotate part of the filtered texts to train a new fastText classifier,
and conduct a second filtering, resulting in the finer filtered math texts. Then we use an instruction-
tuned LLM to extract reasoning steps from these math-related texts, and translate the reasoning steps
into corresponding code snippets. We execute the code snippets and compare the output with the
expected result. If the code executes successfully and the result is as expected, the code is retained.

2.1 BASIC DATA

We collect and carefully filter a diverse range of mathematical data to ensure relevance and quality
for continued pretraining of LLMs. The data includes math-related web content, synthetic data, code
utilizing mathematical packages, and mathematical textbooks.

Math-related Web Data. Web data offers a broad range of real-world mathematical examples.
We start with the OpenWebMath (Paster et al., 2023) dataset, which contains mathematical web
pages sourced from Common Crawl. Observing that a significant portion of these documents are
unrelated to mathematics, we instruct the Mixtral-8x7B-Instruct model with a carefully designed
prompt (detailed in Appendix A) to filter out irrelevant texts. Examples of irrelevant texts are shown
in Appendix D. This reduces the dataset from 13.7B tokens to 4.8B tokens (measured using the
Llama-3 tokenizer). We call this filtered version filtered-OpenWebMath.

To further expand the dataset, we train a fastText classifier (Joulin et al., 2016) using filtered-
OpenWebMath as positive samples and random Common Crawl data as negative samples (training
details are explained in Appendix. B). This model helps identify additional math-related documents
within the Common Crawl data from Matrix (Zhang et al., 2024), a general pretraining dataset. A
second round of filtering is performed, where Mixtral-8x7B-Instruct annotates a portion of these
documents, and a new fastText classifier trained based on these annotations further refines the data.
This produces 6.4B tokens, which we label as filtered-CC-En-math. Finally, we combine filtered-
OpenWebMath and filtered-CC-En-math, resulting in a comprehensive 11.2B-token math-related
web dataset.

Synthetic Data. Synthetic data offers structured mathematical texts that complement the web data.
We collect synthetic data from various open-source repositories on Hugging Face, including datasets
like Education-College-Students1, Maths-College2, and synthetic math books from Matrix (Zhang

1https://huggingface.co/datasets/ajibawa-2023/Education-College-Students
2https://huggingface.co/datasets/ajibawa-2023/Maths-College

3

(b) MathCoder 2.0(a) Other MethodsCommonCrawlRule-based or ModelsMath-relatedTextsCommonCrawlFiner Filtered Math TextsfastTextfastTextInitial  Filtered Math TextsFiner Filtered Math Texts:Conditions Needed:1.The integral is taken over a 3D region defined by the limits of integration.2.The integrand is a function of three variables: 𝜌,∅, and 𝜃Computation Expression:8න0𝜋2න0𝜋2න01𝜌2sin∅𝑑𝜌𝑑𝜃𝑑∅Computation Result:4𝜋3Python Code Snippet:importnumpyasnpfromscipy.integrateimporttplquaddefintegrand(rho, theta, phi):returnrho**2*np.sin(phi)result, _ =tplquad(integrand, 0, np.pi/2, lambdaphi: 0, lambdaphi: np.pi/2, lambdaphi, theta: 0, lambdaphi, theta: 1)print(result *8) # multiply by 8 to match the original expression……𝑥=𝑟sin(∅)cos(𝜃)𝑦=𝑟sin(∅)sin(𝜃)𝑧=𝑟cos(∅)To calculate the volume:𝑣𝑜𝑙=8න0𝜋2න0𝜋2න01𝜌2sin∅𝑑𝜌𝑑𝜃𝑑∅=4𝜋3……Translation LLMInterleaved Reasoning Steps and Code SnippetsExecute Code and Compare with ResultDeepseek mathLemma… Under review as a conference paper at ICLR 2025

Prompt: You will be presented with a text related to math. I need you to identify all the complex
computations in it. For each complex computation, find out the conditions needed for the computation,
the LaTeX expression that conducts the computation, and the result of the computation. Then generate
a Python code snippet for each computation that demonstrates how the result is reached. Output each
computation in the following format:

Conditions Needed:
1. [Condition 1]
2. [Condition 2]
...

Computation Expression:
$[LaTeX Expression]$

Computation Result:
[Computation Result]

Python Code Snippet:

‘‘‘python

[Python Code]
‘‘‘

There can be more than one complex computation in the text. Output only the computations that
requires calculation. Do not include mathematical statements or definitions as a computation. Make
sure each snippet can be executed individually. The text is as follows: {TEXT}
The computations are:

Figure 2: The prompt for extracting reasoning steps from texts in the pretraining corpus and generat-
ing the corresponding Python snippets. {TEXT} is replaced with the text from the dataset collected
in Sec. 2.1.

et al., 2024). To ensure relevance, we apply a fastText classifier to filter out non-mathematical
documents, refining the dataset to 2.2B tokens of high-quality synthetic math content.

Code Utilizing Mathematical Packages. Code data offers practical examples of how mathematical
libraries are used in programming. We collect code from Python and Jupyter files within the Star-
CoderData dataset (Li et al., 2023), retaining only programs that import math-related packages such
as sympy, fractions, cmath, scipy, or statistics. The widely used numpy package is not used to filter
the documents, as it appears frequently in non-mathematical contexts. After filtering, this collection
process results in 1.7B tokens of code related to mathematical computations.

Mathematical Textbooks. Textbooks provide formal, structured presentations of mathematical con-
cepts, making them a valuable source of math knowledge. We gather 8K PDFs of textbooks from
online resources by identifying those with titles containing math-related keywords such as algebra,
geometry, probability, etc. These PDF files are then converted into markdown format using the
Nougat tool for easier integration into our training pipeline.

2.2 MODEL-TRANSLATED MATHEMATICAL CODE

In this section, we propose a novel approach for extracting reasoning steps from the basic pretrain-
ing data and translating them into corresponding Python code snippets that capture the underlying
reasoning processes. This extraction and translation process is performed using a strong instruction-
tuned model, which is Llama-3.1-70B-Instruct in this paper.

Our method begins with taking a piece of text from the basic pretraining data and wrapping it in a
carefully designed prompt, as shown in Fig. 2. This prompt instructs the model to identify LaTeX
expressions denoting complex computations, along with the necessary context, including the condi-
tions required for the computation and the expected result. By explicitly extracting the conditions of
the LaTeX expression, we enhance the model’s ability to comprehend the underlying mathematical
reasoning behind the usage of the expression. The expected result of the computation can later serve
as a basis for verifying the correctness of the generated code. A mathematical reasoning step is con-

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Table 1: The components and data statistics of MathCode-Pile.

Components

Size (MB)

Documents

Tokens

Average (Tokens)

Filtered-OpenWebMath
Filtered-CC-En-math
Synthetic data
Code using math packages
Mathematical textbooks
Translated mathematical code

Total

16,999
23,465
8,855
6,120
4,431
8,235

68,105

2,824,705
7,597,718
2,195,974
513,059
8,373
6,347,823

4,826,902,621
6,341,745,645
2,193,189,314
1,703,226,005
1,390,268,773
2,728,740,985

1,709
835
999
3,320
166,042
430

19,487,652

19,184,073,343

984

structed by combining the conditions, expression and result. The prompt then directs the model to
produce a Python code snippet that accurately reflects the underlying reasoning process behind the
extracted reasoning step. The model is asked to present the conditions, LaTeX expression, result,
and Python code snippet in a structured format, ensuring that each part can be easily extracted from
the generated text. Examples of generated texts are shown in Appendix C.

After the Python code snippets are generated, they are executed, and outputs of the execution are
compared with the expected results extracted from the generated text. Only the Python code snippets
that execute without errors and produce correct outputs are retained. This filtering process ensures
a higher quality of generated code, making the resulting dataset more reliable for mathematical
pretraining compared to approaches that rely on unverified and general-purpose code.

Leveraging the Llama-3.1-70B-Instruct model, we initially generated 3.1B tokens of the data. After
applying the filtering process, we obtain a total of 2.7B tokens of high-quality data of mathemati-
cal code snippets accompanied with their corresponding reasoning steps. This newly generated data
significantly enriches our original pretraining corpus. By combining this data with the basic pretrain-
ing data, we create a comprehensive pretraining dataset totaling 19.2B tokens, which we refer to as
MathCode-Pile. Detailed statistics of MathCode-Pile are presented in Tab. 1. This dataset is tai-
lored specifically for enhancing the mathematical and coding abilities of LLMs. To avoid benchmark
contamination, we follow Yang et al. (2024b) to filter out samples that have significant overlaps with
any of the questions from benchmark datasets used in evaluation. We use exact match to remove
the identical samples and further apply 13-gram deduplication (with a condition that the Jaccard
similarity should be larger than 0.6) to remove more samples that might cause contamination.

In comparison to traditional methods of curating math-related code, which often draw on general-
purpose repositories, our method ensures that the code is not only syntactically correct but also
mathematically sound, reflecting a deeper understanding of mathematical reasoning. Our math-
ematical code is accompanied with corresponding natural language reasoning steps, which makes
understanding the reasoning process easier. This makes MathCode-Pile a superior resource for mod-
els aimed at performing advanced mathematical reasoning tasks.

3 EXPERIMENTS

To demonstrate the effectiveness of our method, we first train several base models ranging from 7B
to 8B parameters using MathCode-Pile and compare them to other best-performing models of the
same size. The group of models resulting from the continued mathematical pretraining is named
MathCoder2. Next, we train and compare various other open-source math pretraining datasets
against MathCode-Pile using a smaller model, DeepSeekCoder-1.3B. To showcase the potential of
the MathCoder2 models, we further perform supervised fine-tuning on them. Finally, we conduct
ablation studies to analyze the impact of each component of the dataset.

3.1 MAIN RESULTS

Benchmark datasets. We evaluate the MathCoder2 models on five representative datasets:
GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), SAT-Math (Azerbayev et al., 2024),
OCW (Lewkowycz et al., 2022), and MMLU-Math (Hendrycks et al., 2021a). GSM8K and MATH

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Table 2: Performance of various pretrained models on five representative mathematical datasets.
All results reported are based on greedy decoding. “Code-open” shows whether the code for data-
processing and model-training is open-sourced. The red numbers show the improvements compared
to the base model from which each MathCoder2 model is trained.
MATH GSM8K

Model

SAT

Size

Code-
open

OCW MMLU-
MATH

Qwen2-Math
Qwen2.5-Math
InternLM2.5
InternLM2-Math-Base
Llemma
Llama-2

Llama-3
MathCoder2-Llama-3

DeepSeekMath
MathCoder2-DeepSeekMath

Mistral
MathCoder2-Mistral

Code-Llama
MathCoder2-Code-Llama

7B
7B
7B
7B
7B
7B

8B
8B

7B
7B

7B
7B

7B
7B

✗
✗
✗
✗
✓
✗

✗
✓

✗
✓

✗
✓

✗
✓

50.4
55.4
34.0
21.5
18.0
3.2

21.4

80.4
91.6
74.8
49.2
36.4
11.8

54.8

87.5
-
65.6
-
53.1
25.0

56.3

14.0
-
8.1
-
7.7
3.7

10.3

38.4(+17.0) 69.9(+15.1) 84.4(+28.1) 18.0(+7.7)

36.2
38.6(+2.4)

64.2
68.8(+4.6)

84.4
90.6(+6.2)

13.1

52.2
36.7(+23.6) 68.2(+16.0) 81.3(+6.3)

75.0

6.7

14.6
28.8(+22.1) 52.3(+37.7) 71.9(+46.9)

25.0

15.4
16.9(+1.5)

8.5
13.2(+4.7)

3.7
8.5(+4.8)

57.9
-
49.6
-
-
-

42.8
46.5(+3.7)

47.4
48.3(+0.9)

38.3
42.2(+3.9)

26.4
33.7(+7.3)

are tested using a 4-shot prompt with MAmmoTH’s evaluation framework (Yue et al., 2023). SAT-
Math and OCW are tested using a 4-shot prompt with DeepSeekMath’s evaluation framework (Shao
et al., 2024). MMLU-Math is tested using the lm-evaluation-harness’s (Gao et al., 2024) default
zero-shot setting for MMLU. These datasets cover a wide range of mathematical problems across
various types and difficulty levels, from primary school math word problems to college-level chal-
lenges, providing a comprehensive evaluation of the models.

Base models and training settings. To demonstrate that our pretraining corpus is effective across
different base models, we continue pretraining four base models with MathCode-Pile: Llama-3-
8B (Dubey et al., 2024), DeepSeekMath-7B (Shao et al., 2024), Mistral-7B (Jiang et al., 2023),
and Code-Llama-7B (Rozi`ere et al., 2024). MathCoder2-Llama-3-8B is trained for 3 epochs with a
global batch size of 4 million tokens and an 8192 token context length. MathCoder2-DeepSeekMath,
MathCoder2-Mistral, and MathCoder2-CodeLlama are each trained for 3 epochs with a global batch
size of 4 million tokens and a 4096 token context length.

Baselines. We compare our method with various other base models that possess strong mathemat-
ical abilities and are of similar sizes, including Qwen2-Math 7B (Yang et al., 2024a), Qwen2.5-
Math 7B (Yang et al., 2024b), InternLM2-Math-Base 7B (Ying et al., 2024), InternLM2.5 7B (Cai
et al., 2024), DeepSeekMath 7B (Shao et al., 2024), Llemma 7B (Azerbayev et al., 2024), Mistral
7B (Jiang et al., 2023), Llama2 7B (Touvron et al., 2023), Llama3 8B (Dubey et al., 2024) and
Code-Llama 7B (Rozi`ere et al., 2024).

Results: As demonstrated in Tab. 2, continued pretraining on MathCode-Pile consistently improves
performance across all five benchmark datasets. MathCoder2 models rival the performance of top
models like InternLM2-Math-Base, InternLM2.5, and DeepSeekMath. In particular, MathCoder2-
DeepSeekMath demonstrates that our method continues to enhance the performance of DeepSeek-
Math, a model that has already been extensively trained on large amounts of math-related data. How-
ever, there remains a performance gap between MathCoder2 and the Qwen2-Math and Qwen2.5-
Math models. This gap might be attributed to their superior computational, manual, and financial
resources, which enable the scaling of data size and the further improvement of data quality, report-
ing a mathemtaical dataset of 700B tokens (Yang et al., 2024b).

In contrast to models like Qwen2-Math, which only open-source their model weights, with much of
their data processing and training details undisclosed, MathCoder2 is fully open-sourced, including
all data processing pipelines and training code. This openness facilitates transparency, reproducibil-

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 3: Performance of various finetuned models on five representative mathematical datasets. All
results reported are based on greedy decoding.

Model

Size MATH GSM8K OCW Olympiad

SVAMP

Qwen2-Math-Instruct
Qwen2.5-Math-Instruct
DeepSeekMath-Instruct-CoT
DeepSeekMath-Instruct-TIR
InternLM2-math-plus
NuminaMath-7B-CoT
NuminaMath-7B-TIR
ToRA-Code
MathCoder
MAmmoTH2-Plus
Llama-3.1-Instruct

MathCoder2-Llama-3-Instruct-CoT
MathCoder2-Llama-3-Instruct-TIR
MathCoder2-DeepSeekMath-Instruct-CoT
MathCoder2-DeepSeekMath-Instruct-TIR

7B
7B
7B
7B
7B
7B
7B
7B
7B
8B
8B

8B
8B
7B
7B

75.1
83.6
46.8
57.4
54.4
55.2
68.1
44.6
30.2
42.8
47.2

58.5
69.7
55.2
69.6

89.9
95.2
82.9
83.7
84.0
75.4
84.6
72.6
67.8
84.1
76.6

83.9
85.8
80.3
86.5

34.6
37.1
-
-
17.3
19.1
-
-
-
-
21.7

29.4
37.6
30.9
41.9

Bench

38.2
41.6
-
-
18.8
19.9
-
-
-
-
15.4

25.8
37.6
23.0
37.9

-
-
-
-
-
-
-
70.4
70.7
-
-

92.7
94.9
92.1
92.8

ity, and further research, which is crucial for advancing the field. Compared to Llemma, which also
open-sources its code, our method achieves better performance on the five datasets. Particularly,
when trained on the same base model, Code-Llama, our method performs significantly better, which
demonstrates the effectiveness of the MathCode-Pile pretraining corpus.

3.2 POST-TRAINING

To further demonstrate the potential of the MathCoder2 models in aligning to mathematical problem-
solving tasks, we select the MathCoder2-Llama-3-8B model and MathCoder2-DeepSeekMath-7B
for finetuning on mathematical problem-solution pairs. We first train the base model on general
mathematical instructions following Yue et al. (2024) for two epochs. Subsequently, we finetune the
model on NuminaMath-CoT3, and NuminaMath-TIR4 datasets for three epochs.

The results are shown in Tab. 3. MathCoder2-Instruct-TIR achieves high performance on all five
datasets, reaching 69.7% on MATH and 86.5% on GSM8K, outperforming many of the best open-
source models of similar size and demonstrating our method’s potential to improve performance
on downstream mathematical reasoning tasks. As this paper focuses on continued mathematical
pretraining, the post-training serves only as a validation of the potential of our models. We conducted
only simple supervised fine-tuning, without performing reinforcement learning or direct preference
optimization, which could further improve performance on downstream tasks.

3.3 ABLATION STUDIES

In this session, we first analyze the impact of various components of the training data. Next, we
compare MathCode-Pile to other open-source mathematical pretraining corpora.

Analysis of the impact of the mathematical code. We analyze the impact of the mathematical code
on continued pretraining by comparing the results of adding and not adding the mathematical code.
As shown in Tab. 4, the addition of the mathematical code in the pretraining corpus significantly
improves performance across all five datasets. Note that the mathematical code only constitutes
14.1% of the 19.2B tokens in the MathCode-Pile dataset, yet the improvement in accuracy it brings
about compared to the total improvement in accuracy ( accMathCode-Pile−accbasic
) is 21.8%, 27.1%, 44.5%,
accMathCodePile−accorig
66.2%, and 35.1% on the five benchmark datasets, respectively, demonstrating the effectiveness of
the mathematical code. Comparison across different training steps is shown in Appendix F.

3https://huggingface.co/datasets/AI-MO/NuminaMath-CoT
4https://huggingface.co/datasets/AI-MO/NuminaMath-TIR

7

Under review as a conference paper at ICLR 2025

Table 4: Analysis of the impact of the mathematical code. The upper half presents the results of
using and not using the mathematical code data. The lower half analyzes design of concatenating
the reasoning steps and code snippets. “Basic + Reasoning-step-only” represents only adding the
conditions, expressions, and results, while “Basic + Trans-code-only” represents only adding the
translated code. “Basic + Separated Text&Code” represents seperating corresponding code and
text. “Reasoning-Step&Code” represents the concatenated data combining both. “Basic + No-code-
prompt” represents using a prompt that simply instruct Llama-3.1-70B-Instruct to rewrite texts to
improve their quality.

Data Composition

Base Model

MATH GSM8K SAT

OCW MMLU-

MATH

Basic
Basic + Reasoning-Step&Code

Llama-3-8B
Llama-3-8B

34.7
71.9
38.4(+3.7) 69.9(+4.1) 84.4(+12.5) 18.0(+5.1) 46.5(+1.3)

12.9

65.8

45.2

Basic + Reasoning-step-only
Basic + Trans-code-only
Basic + No-code-prompt
Basic + Separated Text&Code
Basic + Reasoning-Step&Code

DeepSeekCoder-1.3B
DeepSeekCoder-1.3B
DeepSeekCoder-1.3B
DeepSeekCoder-1.3B
DeepSeekCoder-1.3B

16.7
14.6
15.7
17.0
17.8

22.7
22.1
21.3
22.0
25.5

40.6
43.8
37.5
46.9
59.4

4.8
5.5
4.8
4.8
5.9

25.9
25.5
24.4
25.3
26.1

Table 5: Analysis of the effect of different components in MathCoder2-Pile. The base model is
DeepSeekCoder-1.3B.

Data Composition

No Math Training

filtered-OpenWebMath (4.8B)
OpenWebMath (12.9B)

filtered-CC-En-math (6.4B)
CC-En-math (22.1B)

filtered-OpenWebMath + textbooks
filtered-OpenWebMath + synthetic data
filtered-OpenWebMath + code

MathCoder2-Pile

MATH GSM8K SAT

OCW

MMLU-MATH

4.8

9.0
9.4

9.1
8.4

9.4
10.8
9.4

17.8

4.3

11.4
11.2

12.1
13.0

12.7
12.6
12.1

25.5

18.8

34.4
31.3

31.3
25.0

50.0
50.0
46.9

59.4

2.6

3.7
2.6

3.7
2.9

4.0
4.0
4.0

5.9

24.8

25.4
24.4

25.2
25.0

25.4
25.6
25.4

26.1

We also analyze the design choice of concatenating the natural language reasoning step with the
mathematical code for pretraining. This analysis is conducted by studying the results of adding only
the natural language reasoning steps, and separately adding only the translated code. As shown in
Tab. 4, Basic + Reasoning-step-only represents adding only the natural language reasoning steps;
Basic + Trans-code-only represents adding only the translated code; Basic + Separated Text&Code
represents seperating code and text; and Basic + Reasoning-Step&Code represents adding the con-
catenated data that combines both. The Basic + Reasoning-Step&Code configuration results in the
best performance, demonstrating the importance of including both the natural language reasoning
step and the translated mathematical code.

To rule out the possibility that the improvement comes from the higher quality of texts generated
by Llama-3.1-70B-Instruct, we use a prompt that asks Llama-3.1-70B-Instruct to rewrite the given
text. The details of this prompt are provided in Appendix E. We present the results of replacing the
mathematical code with texts generated using this prompt in Tab. 4, labeled as “Basic + No-code-
prompt”. Our method of generating mathematical code accompanied with corresponding reasoning
steps outperforms this baseline, demonstrating the effectiveness of our approach.

Analysis of the impact of various parts of the basic data. We perform experiments on a
smaller model, DeepSeekCoder-1.3B, using different parts of the basic data. As demonstrated in
Tab. 5, filtered-OpenWebMath and filtered-CC-En-math significantly improve the performance of
the model. In comparison, textbooks, synthetic data, and code are smaller in data size and play a
less important role. As each of these parts of data is too small for individual pretraining, we combine

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

Table 6: Comparison between MathCode-Pile and other Mathematical Pretrain datasets.

Pretrain Dataset

Base Model

MATH GSM8K SAT

OCW MMLU-MATH

No Math Training

DeepSeekCoder-1.3B

OpenWebMath
Proof-Pile-2
MathPile
DeepSeekMath Corpus

DeepSeekCoder-1.3B
DeepSeekCoder-1.3B
DeepSeekCoder-1.3B
DeepSeekLLM-1.3B

MathCoder2-Pile

DeepSeekCoder-1.3B

4.8

9.4
9.2
5.3
13.6

17.8

4.3

11.2
11.2
3.4
23.8

25.5

18.8

31.3
50.0
21.9
56.3

59.4

2.6

2.6
4.4
2.2
4.8

5.9

24.8

24.4
25.8
24.9
-

26.1

Table 7: Comparison between finetuning the original Llama-3-8B, MathCoder2-Basic-Llama-3-8B,
and MathCoder2-Llama-3-8B on NuminaMath-TIR. MathCoder2-Basic-Llama-3-8B is the model
resulting from continued pretraining on the basic data without adding the model-translated mathe-
matical code.
Base Model

GSM8K

SVAMP

MATH

OCW

Olympiad
Bench

Llama-3-8B
MathCoder2-Basic-Llama-3-8B
MathCoder2-Llama-3-8B

56.1
62.9
65.1

80.1
81.3
84.5

24.6
26.8
34.6

28.4
32.9
34.4

83.8
86.7
87.9

them with OpenWebMath-filtered to show that they each bring a small yet noticeable improvement
compared to using only OpenWebMath-filtered. Since we performed filtering on OpenWebMath
and the initially filtered CC-En to remove irrelevant data, we also compare the performance before
and after filtering. We observe that there is no obvious degradation in performance after removing
irrelevant content, showing the effectiveness of the filtering.

Comparison with other open-source mathematical pretraining corpora. We compare
MathCode-Pile with various other open-source mathematical pretraining corpora. We train each
corpus for 3 epochs with a global batch size of 2 million tokens and a 4096 token context length,
since we observe that the model’s performance usually saturates around 3 epochs. As shown in
Tab. 6, MathCode-Pile significantly outperforms OpenWebMath, Proof-Pile-2, and MathPile when
trained on DeepSeekCoder-1.3B. The DeepSeekMath Corpus is not open-source, and its perfor-
mance on DeepSeekLLM-1.3B is taken from Shao et al. (2024), which is trained for 150B tokens,
more than our MathCode-Pile’s training of approximately 60B tokens. The 1.3B model trained with
MathCode-Pile outperforms the 1.3B model trained with DeepSeekMath Corpus.

Analysis of the improvement on the potential of being finetuned for TIR reasoning. To analyze
the effect of the model-translated mathematical code on LLMs’ potential to be finetuned for TIR
reasoning, we finetune the original Llama-3-8B, MathCoder2-Basic-Llama-3-8B, and MathCoder2-
Llama-3-8B on NuminaMath-TIR for three epochs, respectively. As shown in Tab. 7, the results
of finetuning on MathCoder2-Basic-Llama-3-8B are higher than the results of finetuning on Llama-
3-8B. Finetuning on MathCoder2-Llama-3-8B results in even higher performance than finetuning
on MathCoder2-Basic-Llama-3-8B, showing that the addition of mathematical code effectively en-
hances the models’ potential of being finetuned for TIR reasoning.

4 RELATED WORK

Continued mathematical pretraining. Several works (Shao et al., 2024; Azerbayev et al., 2024;
Ying et al., 2024; Yang et al., 2024b) have explored the continued pretraining of LLMs on math-
ematical data, such as mathematical web content, synthetic data, and code. InternLM-Math (Ying
et al., 2024) and Query of CC Fei et al. (2024) use BM25 for data retrieval, while other works such
as DeepSeekMath (Shao et al., 2024) and Qwen2-Math (Yang et al., 2024b) employ fastText (Joulin
et al., 2016) and other meta-information to retrieve texts from Common Crawl. Our approach fol-
lows these methods by using fastText for data filtering, and we introduce a second iteration of finer
filtering to retain more relevant data. MathPile (Wang et al., 2023b) and phi (Gunasekar et al.,

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

2023) utilize real or synthesized textbooks, while Llemma (Azerbayev et al., 2024) and Qwen2-
Math (Yang et al., 2024b) incorporate math-related code in their datasets. However, unlike our
method of generating mathematical code with accompanied natural language reasoning, their code
mostly has no natural language explanations or context. Our work builds on these prior efforts by
collecting and expanding upon these sources of math-related text. Unlike works that only open-
source their model weights, we take a more transparent approach by open-sourcing both our data
processing and model training code, thereby ensuring reproducibility and facilitating future research
in this field. Compared to Llemma (Azerbayev et al., 2024), which also open-source their data and
training code, our method results in better performance on mathematical reasoning tasks.

Synthetic data. Numerous finetuning (Yu et al., 2024; Wang et al., 2023a; Lu et al., 2024a) and pre-
training Gunasekar et al. (2023); Wang et al. (2023b); Yang et al. (2024b) studies have explored train-
ing on synthetic data generated using language models or predefined templates. MathGLM (Yang
et al., 2023) and InternLM-Math (Ying et al., 2024) use templates to generate synthetic numeri-
cal operation data, while phi (Gunasekar et al., 2023) produces textbook-quality data with models.
EntiGraph (Yang et al., 2024c) generates diverse text by drawing connections between sampled enti-
ties. Our work proposes a novel method for extracting mathematical reasoning steps and generating
synthetic code snippets that captures the underlying reasoning processes.

Post-training. There are many methods for further improving the mathematical problem-solving
abilities of LLMs. Supervised finetuning adjusts pretrained models using math problems and solu-
tions in various formats, such as Chain-of-Thought (Yu et al., 2024; Yuan et al., 2023), Program-
of-Thought (Yue et al., 2023), and Tool-Integrated Reasoning (Gou et al., 2024; Wang et al., 2023a;
Liao et al., 2024). Reinforcement learning Lightman et al. (2023); Wang et al. (2024) and Direct
Preference Optimization Rafailov et al. (2024); Xu et al. (2024); Lu et al. (2024b) utilize mathemati-
cal preference data to adjust the models’ outputs. These methods are diverse and reveal the potential
of pretrained models. Their performance is often influenced by the quality of the training data used
in the pretraining stage. To explore the potential of finetuning our pretrained models for downstream
tasks, we conduct supervised finetuning with existing open-source data.

5 LIMITATIONS AND FUTURE WORK

One limitation of our work is that our continued pretraining corpus focuses primarily on mathe-
matics and does not intentionally include other STEM subjects. Additionally, our pretraining data
consists entirely of English texts, without incorporating math-related content in other languages, like
Chinese. Due to limitations in computational resources, we only trained models ranging from 1.3B
to 8B parameters. Future work could address these limitations by expanding the dataset to include
other subjects and languages and by training on larger language models. Also, this paper primarily
focuses on continued mathematical pretraining, so we did not apply reinforcement learning meth-
ods like PPO and GRPO, or Direct Preference Optimization in our post-training phase, which can
further improve performance on mathematical reasoning tasks. In the future, we could explore these
methods on our finetuned models. Also, this work did not discuss theorem proving with formal
languages such as Lean and Coq, which is worth investigating in future works.

6 CONCLUSION

In this paper, we present an effective open-source continued mathematical pretraining pipeline for
enhancing mathematical reasoning of LLMs. Through the meticulous collection and filtering of di-
verse math-related texts, such as mathematical web content, synthetic data, code that uses mathemat-
ical packages, and math textbooks, we curate a basic dataset for continued mathematical pretraining.
We then propose a novel method for extracting mathematical reasoning steps from the previously
collected dataset and translating them to code snippets reflecting the underlying reasoning processes.
By combining the basic data with the model-generated mathematical code accompanied with their
corresponding reasoning steps, we produce a 19.2B-token mathematical pretraining corpus named
MathCode-Pile, which significantly improves the performance of four different base models across
five representative mathematical benchmarks. By open-sourcing the entire data processing pipeline
and model training code, we actively promote transparency, reproducibility, and collaboration within
the research community, facilitating future research in this area.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al-
bert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model
for mathematics, 2024. URL https://arxiv.org/abs/2310.10631.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about
physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.
11641.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui
Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye
Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting
Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li,
Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun
Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang
Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song,
Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang,
Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong,
Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia
Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo
Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui
Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou,
Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John
Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.
org/abs/2110.14168.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.

Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Conghui He, Hang Yan, Dahua Lin, and
Xipeng Qiu. Query of cc: Unearthing large scale domain-specific knowledge from public corpora,
2024. URL https://arxiv.org/abs/2401.14624.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos-
ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muen-
nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin-
tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework
for few-shot language model evaluation, 07 2024. URL https://zenodo.org/records/
12608602.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and
Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, 2024.
URL https://arxiv.org/abs/2309.17452.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth
Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital
Shah, Harkirat Singh Behl, Xin Wang, S´ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai,
Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv.org/
abs/2306.11644.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja-
cob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https:
//arxiv.org/abs/2009.03300.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b.
URL https://arxiv.org/abs/2103.03874.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap-
lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier,
L´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril,
Thomas Wang, Timoth´ee Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv.org/abs/2310.06825.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient

text classification, 2016. URL https://arxiv.org/abs/1607.01759.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra-
masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam
Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan-
guage models, 2022. URL https://arxiv.org/abs/2206.14858.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao
Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii,
Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, Jo˜ao
Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo-
gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra
Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey,
Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc-
cioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor,
Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex
Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva
Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Mu˜noz Ferrandis, Sean Hughes,
Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source
be with you!, 2023. URL https://arxiv.org/abs/2305.06161.

Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code
interpreter output – a reproducible pipeline, 2024. URL https://arxiv.org/abs/2401.
08190.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL
https://arxiv.org/abs/2305.20050.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by
chatgpt really correct? rigorous evaluation of large language models for code generation, 2023.
URL https://arxiv.org/abs/2305.01210.

Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hong-
sheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing
mathematical reasoning of llms, 2024a. URL https://arxiv.org/abs/2402.16352.

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hong-
sheng Li. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning,
2024b. URL https://arxiv.org/abs/2407.00782.

Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open
dataset of high-quality mathematical web text, 2023. URL https://arxiv.org/abs/
2310.06786.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and
Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model,
2024. URL https://arxiv.org/abs/2305.18290.

Baptiste Rozi`ere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi
Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, J´er´emy Rapin, Artyom Kozhevnikov, Ivan Ev-
timov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong,
Alexandre D´efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier,
Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.
URL https://arxiv.org/abs/2308.12950.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver-
sarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.
10641.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang,
Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe-
matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.
03300.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher,
Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy
Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,
Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi,
Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models,
2023. URL https://arxiv.org/abs/2307.09288.

Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi
Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for en-
hanced mathematical reasoning, 2023a. URL https://arxiv.org/abs/2310.03731.

Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang
Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024.
URL https://arxiv.org/abs/2312.08935.

Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i – mathpile: A billion-token-
scale pretraining corpus for math, 2023b. URL https://arxiv.org/abs/2312.17120.

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan
Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, and Yuxiao Dong. Chatglm-math: Improving math
problem-solving in large language models with a self-critique pipeline, 2024. URL https:
//arxiv.org/abs/2404.02893.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
arXiv preprint

Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.
arXiv:2407.10671, 2024a.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu,
Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu,
Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical ex-
pert model via self-improvement, 2024b. URL https://arxiv.org/abs/2409.12122.

Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang.
Gpt can solve mathematical problems without a calculator, 2023. URL https://arxiv.
org/abs/2309.03241.

Zitong Yang, Neil Band, Shuangping Li, Emmanuel Cand`es, and Tatsunori Hashimoto. Synthetic

continued pretraining, 2024c. URL https://arxiv.org/abs/2409.07431.

Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma,
Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou,
Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen,
and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning,
2024.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Prompt: You will be provided with a block of text. I need you to classify the text into one of the
following types:
1. The text describes a mathematical problem and its solution.
2. The text explains a mathematical concept or mathematical theory.
3. The text explains a scientific or engineering concept that requires mathematical knowledge.
4. The text describes a programming problem and its solution.
5. The text explains a concept or theory related to programming.
6. The text explains the usage of a programming language or software tool.
7. The text does not belong to any of the types above.
Here’s the text I’ve provided. Kindly analyze and classify it into type 1, 2, 3, 4, 5, 6 or 7. Put your
choice behind “The type is:”. Please do not generate any unrelated additional comments! The type
number must match the type description. Here’s one of the texts that needs to be classified: {TEXT}
The type is:

Figure 3: The prompt for annotation of OpenWebMath and the initially filtered CC-En documents.
{TEXT} is replaced with the content of the document.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhen-
guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions
for large language models, 2024. URL https://arxiv.org/abs/2309.12284.

Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou,
and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language
models, 2023. URL https://arxiv.org/abs/2308.01825.

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
Mammoth: Building math generalist models through hybrid instruction tuning, 2023. URL
https://arxiv.org/abs/2309.05653.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the

web, 2024. URL https://arxiv.org/abs/2405.03548.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma-
chine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830.

Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan,
Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming
Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting
Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu,
Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai,
Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli
Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual
large language model series, 2024. URL https://arxiv.org/abs/2405.19327.

Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia,
Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using
GPT-4 code interpreter with code-based self-verification. In The Twelfth International Confer-
ence on Learning Representations, 2024. URL https://openreview.net/forum?id=
c8McWs4Av0.

A PROMPT FOR ANNOTATION OF MATH WEB DOCUMENTS

In this section, we present the prompt we used for annotation of documents in OpenWebMath and
the initially filtered CC-En. The prompt, as shown in Fig. 3, asks the model to classify the document
into one of seven types, which are types of documents that frequently appear in the datasets. We
observe that this method helps the model to better identify and filter out irrelevant text than using a
binary classification of whether the text is related to math.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Prompt: You will be presented with a text related to math. I need you to carefully read through the
text. If you find any incorrect statments, erroneous computation steps, spelling mistakes, grammatical
errors, or formatting issues, adjust them so that the error is corrected. Rewrite the text to make it more
accurate and easier to understand. You should only output an adjusted version of the given text. Also,
do not change the original language. Please do not generate any unrelated additional comments! The
text is as follows: {TEXT}
You should output:

Figure 4: The prompt asking Llama-3.1-70B-Instruct to simply rewrite the text and improve its
quality. {TEXT} is replaced with the content of the document.

B TRAINING DETAILS OF FASTTEXT CLASSIFIERS

We employ an open-source library5 for training, configuring the vector dimension to 50, the learning
rate to 0.5, the maximum length of word n-grams to 2, and the number of training epochs to 5. For
the initial filtering of the Common Crawl corpus, we sample 3 million data points from the seed
corpus of filtered-OpenWebMath as positive training examples and another 8 million web pages
from Common Crawl as negative examples. For finer filtering, we use 2 million data points annotated
as math-related by Mixtral-8x7B-Instruct as positive training samples and 1 million data points
annotated as unrelated to math as negative training samples.

C EXAMPLES OF THE MODEL-GENERATED MATHEMATICAL CODE

This session shows several examples of the translation from math-related texts to mathematical code
acompanied with corresponding reasoning steps. As shown in Tab. 8, Tab. 9, Tab. 10 and Tab. 11,
the model first extract the LaTex expression alone with its conditions and result from the original
text, then generates an Python code snippet based on this information.

D EXAMPLES OF REMOVED IRRELEVANT TEXTS

In this section, we present several examples in the original OpenWebMath dataset that are irrelevant
to mathematical reasoning and removed in the filtering process. As shown in Tab. 12, Tab. 13, and
Tab. 14, the content of these documents are not related to math, but instead are about subjects such
as politics, testing software, or web development. Removing these irrelevant texts have no obvious
impact on the mathematical continued pretraining performance.

E PROMPT FOR SIMPLE REWRITING TO IMPROVE QUALITY FOR ABLATION

STUDY

To rule out the possibility that the improvement results solely from the enhanced quality of the
texts generated by Llama-3.1-70B-Instruct, we designed a prompt asking Llama-3.1-70B-Instruct
to rewrite the text, checking for mistakes in content and format to enhance accuracy and clarity, as
shown in Fig. 4.

F COMPARISON BETWEEN ADDING AND NOT ADDING MATHEMATICAL

CODE

In this section, we present the comparison between adding and not adding mathematical code across
different training steps. The experiments are conducted on Llama-3 8B. As shown in Fig. 5 and
Fig. 6, adding the model-translated mathematical code improves accuracy across different training
steps.

5https://fasttext.cc/

15

Under review as a conference paper at ICLR 2025

Original Text:
Bayes and his Theorem
My earlier post on Bayesian probability seems to have generated quite a lot of readers, so this
lunchtime I thought I’d add a little bit of background.
......
If we ask the question “what is the probability of exactly x successes from the possible n?”, the
answer is given by the binomial distribution:

......

Pn(x|n, p) = C(n, x)px(1 − p)n−x

Translation Output:
Conditions Needed:
1. The number of trials (n) is a positive integer.
2. The probability of success (p) is a real number between 0 and 1.
3. The number of successes (x) is a non-negative integer less than or equal to n.
Computation Expression:
Pn(x|n, p) = C(n, x)px(1 − p)n−x
Computation Result:
The probability of exactly x successes in n independent trials, each with a probability of success
p.
Python Code Snippet:

‘ ‘ ‘ p y t h o n
i m p o r t math

d e f b i n o m i a l d i s t r i b u t i o n ( n , x , p ) :

c o m b i n a t i o n s = math . comb ( n , x )
p r o b a b i l i t y = c o m b i n a t i o n s * ( p ** x ) * ( ( 1 − p ) ** ( n − x ) )
r e t u r n p r o b a b i l i t y

# number o f
# number o f

# Example u s a g e :
n = 10
x = 5
p = 0 . 5
r e s u l t = b i n o m i a l d i s t r i b u t i o n ( n , x , p )
p r i n t ( r e s u l t )
‘ ‘ ‘

t r i a l s
s u c c e s s e s

# p r o b a b i l i t y o f

s u c c e s s

Table 8: An example of translating a piece of text to mathematical code and a corresponding rea-
soning step.

G REVISIONEFFECT OF MATHCODE-PILE ON GENERAL CODING AND

OTHER ABILITIES

In order to explore MathCode-Pile’s effect on general coding abilities, we test the MathCoder2 mod-
els on HumanEval and MBPP, two representative coding benchmarks, using the EvalPlus6 frame-
work. HumanEval+ and MBPP+ are extended versions of HumanEval and MBPP with additional
test samples, as introduced in Liu et al. (2023). The pass@1 accuracies are shown in Tab. 15.
MathCode-Pile improves the performance of Llama3-8B, DeepSeekMath-7B, and Mistral-7B on
these coding benchmarks. The performance of MathCoder2-CodeLlama-7B on these benchmarks is
similar to that of CodeLlama-7B, which is understandable since CodeLlama is specifically trained
for code generation. This demonstrates that MathCode-Pile can enhance general coding abilities
beyond math-focused tasks for models not explicitly trained on code.

6https://github.com/evalplus/evalplus

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Original Text:
# Probability of getting 2 Aces, 2 Kings and 1 Queen in a five card poker hand (Part II)
So I reworked my formula in method 1 after getting help with my original question ......

(cid:1)
(cid:1)(cid:0)4
(cid:1)(cid:0)4
(cid:0)4
1
2
2
(cid:1) =
(cid:0)52
5

3
54145

5.540678 ∗ 10−5

......

Translation Output:
Conditions Needed:
1. The total number of cards in a deck is 52.
2. We are drawing 5 cards from the deck.
3. We want to calculate the probability of getting 2 Aces, 2 Kings, and 1 Queen.
Computation Expression:

(cid:1)
(cid:1)(cid:0)4
1

(cid:1)(cid:0)4
(cid:0)4
2
2
(cid:1)
(cid:0)52
5

5.540678 ∗ 10−5

Computation Result:

Python Code Snippet:

‘ ‘ ‘ p y t h o n
i m p o r t math

d e f c o m b i n a t i o n ( n , k ) :

r e t u r n math . comb ( n , k )

n u m e r a t o r = c o m b i n a t i o n ( 4 , 2 ) * c o m b i n a t i o n ( 4 , 2 ) * c o m b i n a t i o n ( 4 , 1 )
d e n o m i n a t o r = c o m b i n a t i o n ( 5 2 , 5 )

r e s u l t = n u m e r a t o r
p r i n t ( r e s u l t )
‘ ‘ ‘

/ d e n o m i n a t o r

Table 9: An example of translating a piece of text to mathematical code and a corresponding rea-
soning step.

To demonstrate how MathCode-Pile affects LLMs’ general abilities, we tested the MathCoder2
models on Hellaswag (Zellers et al., 2019), PIQA Bisk et al. (2019), and Winogrande Sakaguchi
et al. (2019) using the lm-evaluation-harness7. As shown in the table below, training on MathCode-
Pile slightly impacts the performance of general-purpose models, such as Llama3-8B and Mistral-
7B, on these datasets, likely because MathCode-Pile consists entirely of math-related data. The
effect on special-purpose models, such as DeepSeekMath-7B and CodeLlama-7B, is minimal.

H N-GRAMS OVERLAP BETWEEN TRAINING DATA AND TESTING DATA

We also apply n-gram testing to check for the overlapping ratio between MathCode-Pile and the test
datasets, demonstrate the effectiveness of our decontamination. The overlaps at various n-grams are
very low. The overlap becomes 0.00% when n-grams is 13.

7https://github.com/EleutherAI/lm-evaluation-harness

17

Under review as a conference paper at ICLR 2025

Original Text:
# How do you use the product rule to differentiate g(x) = (x2 + 1)(x2 − 2x)?
......
and h (x) = x2 − 2x ⇒ h′ (x) = 2x − 2
......

Translation Output:
Conditions Needed:
1. h(x) is a polynomial function
2. The derivative of x2 is 2x
3. The derivative of −2x is −2
Computation Expression:
d
dx (x2 − 2x) = 2x − 2
Computation Result:
h′(x) = 2x − 2
Python Code Snippet:

‘ ‘ ‘ p y t h o n
i m p o r t

sympy a s

s p

x = s p . s y m b o l s ( ’ x ’ )
h = x **2 − 2* x
h p r i m e = s p . d i f f ( h , x )
p r i n t ( h p r i m e )
‘ ‘ ‘

# O u t p u t : 2* x − 2

Table 10: An example of translating a piece of text to mathematical code and a corresponding
reasoning step.

Figure 5: Comparison of the accuracy on GSM8K between adding and not adding mathematical
code across different training steps.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

010002000300040005000600070008000900010000110001200013000Steps55.057.560.062.565.067.570.0Accuracy (%)GSM8Kbasic datacode-addedUnder review as a conference paper at ICLR 2025

Original Text:
# How do you use the product rule to differentiate g(x) = (x2 + 1)(x2 − 2x)?
......
and h (x) = x2 − 2x ⇒ h′ (x) = 2x − 2
......

Translation Output:
Conditions Needed:
1. h(x) is a polynomial function
2. The derivative of x2 is 2x
3. The derivative of −2x is −2
Computation Expression:
d
dx (x2 − 2x) = 2x − 2
Computation Result:
h′(x) = 2x − 2
Python Code Snippet:

‘ ‘ ‘ p y t h o n
i m p o r t

sympy a s

s p

x = s p . s y m b o l s ( ’ x ’ )
h = x **2 − 2* x
h p r i m e = s p . d i f f ( h , x )
p r i n t ( h p r i m e )
‘ ‘ ‘

# O u t p u t : 2* x − 2

Table 11: An example of translating a piece of text to mathematical code and a corresponding
reasoning step.

## Avoiding Weimar Russia
Matthew Yglesias writes:
Matthew Yglesias: Beyond Economics: Over at Brad DeLong’s site you can see a fascinat-
ing discussion of America’s Russia policy in the 1990s between DeLong, Martin Wolf, and
Lawrence Summers. One remark I would make is that to an extraordinary extent, all three par-
ticipants are willing to accept the premise that the only goal of US policy toward Russia in the
1990s was a good-faith effort to induce Russian prosperity, with such efforts being hampered
by political constraints, the objective difficulty of the task, and pure policy errors...
Well, yes. Russia was once a superpower and may be one again. One would have thought that
the history of 1914-1945 would teach ample lessons about the national security undesirability
of trying to keep great powers–like Weimar Germany–poor and weak. One would have thought
that the history of 1945-1990 would teach ample lessons about the national security desirability
of trying to help great powers–like Japan and West Germany–become prosperous, democratic,
and well-integrated into the world economy.
One top of the national-security strategic argument there is the economic argument: the fact
that richer trading partners are better trading partners: they make more and more interesting
stuff for us to buy.
......

Table 12: An example of removed text irrelevant to mathematical reasoning in OpenWebMath.

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

# MicroEJ Test Suite Engine¶
## Introduction¶
The MicroEJ Test Suite Engine is a generic tool made for validating any development project
using automatic testing.
This section details advanced configuration for users who wish to integrate custom test suites
in their build flow.
The MicroEJ Test Suite Engine allows the user to test any kind of projects within the configu-
ration of a generic Ant file.
The MicroEJ Test Suite Engine is already pre-configured for running test suites on a MicroEJ
Platform (either on Simulator or on Device).
## Using the MicroEJ Test Suite Ant Tasks¶
Multiple Ant tasks are available in the testsuite-engine.jar provided in the Build Kit:
• testsuite allows the user to run a given test suite and to retrieve an XML report document in a
JUnit format.
• javaTestsuite is a subtask of the testsuite task, used to run a specialized test suite for Java (will
only run Java classes).
• htmlReport is a task which will generate an HTML report from a list of JUnit report files.
......

Table 13: An example of removed text irrelevant to mathematical reasoning in OpenWebMath.

By Kimserey Lam with
# Conemu A Better Command Prompt For Windows
Jul 22nd, 2017 - written by Kimserey with .
When developing multiple Web api under multiple Visual Studio solutions, it can become very
tedious to maintain, run and debug. Opening multiple instances of Visual Studio is very costly
in term of memory and running all at once also clutter the screen which rapidly becomes irritat-
ing. With the advent of dotnet CLI tools, it has been clear that the next step would be to move
out of the common “right click/build, F5” of Visual Studio and toward “dotnet run” on a com-
mand prompt. Last month I was looking for a Windows alternative of the bash terminal which
can be found on Mac and I found ConEmu. ConEmu provides access to all typical shells via
an enhanced UI. Today we will see how we can use ConEmu to ease our development process
by leveraging only 2 of its features; the tasks and environment setup.
1. dotnet CLI 2. Setup environment 4. Apply to multiple services
......

Table 14: An example of removed text irrelevant to mathematical reasoning in OpenWebMath.

Table 15: Performance of the MathCoder2 models on general coding benchmarks: HumanEval,
HumanEval+, MBPP and MBPP+, as well as general ability benchmarks: Hellaswag, PIQA and
Winogrande.

Model

Human-
Eval

Human-
Eval+

MBPP MBPP+ Hella-
swag

PIQA Winog-

rande

Llama-3-8B
MathCoder2-Llama-3-8B

DeepSeekMath-7B
MathCoder2-DeepSeekMath-7B

Mistral-7B
MathCoder2-Mistral-7B

Code-Llama-7B
MathCoder2-Code-Llama-7B

40.2
51.8

36.0
36.6

29.3
39.6

37.8
38.4

35.4
43.3

28.7
32.3

23.8
34.1

35.4
32.3

61.9
61.9

64.8
66.7

51.3
54.5

59.5
58.5

52.1
52.1

52.9
54.8

40.5
46.8

46.8
47.4

79.2
75.9

66.4
66.9

81.1
78.1

62.9
62.8

81.0
78.1

74.7
74.0

82.0
78.0

72.5
72.3

73.4
71.7

64.6
63.1

73.9
72.3

64.7
63.7

Table 16: Overlap ratios for different n-grams.

n-grams
Overlap Ratio (%)

3
0.21

4
0.12

5
0.06

6
0.03

7
0.02

8
0.01

13
0.00

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Figure 6: Comparison of the accuracy on MATH between adding and not adding mathematical code
across different training steps.

21

010002000300040005000600070008000900010000110001200013000Steps222426283032343638Accuracy (%)MATHbasic datacode-added