AoPS Dataset: Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
Under review as a conference paper at ICLR 2025

AOPS DATASET: LEVERAGING ONLINE OLYMPIAD-
LEVEL MATH PROBLEMS FOR LLMS TRAINING AND
CONTAMINATION-RESISTANT EVALUATION

Anonymous authors
Paper under double-blind review

ABSTRACT

Advances in Large Language Models (LLMs) have sparked interest in their abil-
ity to solve Olympiad-level math problems. However, the training and evalua-
tion of these models are constrained by the limited size and quality of available
datasets, as creating large-scale data for such advanced problems requires exten-
In addition, current benchmarks are prone to
sive effort from human experts.
contamination, leading to unreliable evaluations.
In this paper, we present an
automated pipeline that leverages the rich resources of the Art of Problem Solv-
ing (AoPS) forum, which predominantly features Olympiad-level problems and
community-driven solutions. Using open-source LLMs, we develop a method
to extract question-answer pairs from the forum, resulting in AoPS-Instruct, a
dataset of more than 650,000 high-quality QA pairs. Our experiments demon-
strate that fine-tuning LLMs on AoPS-Instruct improves their reasoning abili-
ties across various benchmarks. Moreover, we build an automatic pipeline that
introduces LiveAoPSBench, an evolving evaluation set with timestamps, de-
rived from the latest forum data, providing a contamination-resistant benchmark
for assessing LLM performance. Notably, we observe a significant decline in
LLM performance over time, suggesting their success on older examples may
stem from pre-training exposure rather than true reasoning ability. Our work
presents a scalable approach to creating and maintaining large-scale, high-quality
datasets for advanced math reasoning, offering valuable insights into the capa-
bilities and limitations of LLMs in this domain. Our benchmark is available at
https://livemathbench.github.io/leaderboard.

1

INTRODUCTION

Large language models (LLMs) have shown tremendous success in solving various tasks such as
code generation (Li et al., 2022), math reasoning (Shao et al., 2024), and commonsense reason-
ing (Zellers et al., 2019; Achiam et al., 2023), suggesting that current models may show signs of
artificial general intelligence (AGI) (Bubeck et al., 2023). Math reasoning is perhaps one of the
most challenging tasks for the LLMs, since mathematics is inherently structured, requiring not just
recall of facts but also rigorous logical inference, abstraction, and understanding of formal sym-
bolic systems. As such, there have been grand challenges (Selsam et al., 2019) and million-dollar
prizes AIMO (2023) established for a model capable of solving Olympiad-level math problems.

On the training side, despite significant progress in certain areas, such as geometry, particularly
with the assistance of symbolic methods (Trinh et al., 2024), the performance of LLMs remains
limited on Olympiad-level problems (He et al., 2024). One of the key challenges in advancing
competition-level math reasoning, compared to other domains like coding or grade-school math, is
the scarcity of large-scale data. Creating valid and challenging math questions, along with providing
correct solutions, is costly. This is especially true for Olympiad-level problems, which can be time-
consuming even for experts. This highlights the need for scalable and automated methods to collect
high-quality data for Olympiad-level problems to facilitate further advancements in this field.

On the evaluation side, in contrast to the rapid advancements in LLMs, the evaluation of their
math reasoning capabilities remains relatively underdeveloped. First, as aforementioned, the cost

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

of creating and annotating advanced math problems is high. Second, popular datasets such as
MATH (Hendrycks et al., 2021b) and GSM8K (Cobbe et al., 2021) have been saturated by both
open-source and closed-source models (Achiam et al., 2023; Yang et al., 2024b). Third, bench-
marks whose test sets are publicly available online (Hendrycks et al., 2021b; Cobbe et al., 2021; He
et al., 2024; Zhang et al., 2023b) are prone to potential contamination. Although techniques like
n-gram matching and locality-sensitive hashing have been applied as a common practice (Achiam
et al., 2023; Dubey et al., 2024; Yang et al., 2024a) to reduce contamination, they still suffer low ac-
curacy and would not be able to rule out rephrased questions, as shown by Yang et al. (2023). Given
these limitations, it is crucial to develop an evolving evaluation benchmark that contains abundant
and up-to-date test samples, and designed with appropriate difficulty to fairly assess a model’s math
reasoning abilities.

The Art of Problem Solving1 (AoPS) forum is a rich resource for Olympiad-level math problems,
featuring discussions on topics such as algebra, geometry, combinatorics, and number theory from
competitions like AMC (AOPS, 2023), AIME (AOPS, 2024), and the International Mathematical
Olympiad (IMO). However, the forum’s unstructured nature, including irrelevant comments and
incomplete solutions, poses challenges in extracting high-quality, structured question-answer (QA)
pairs. Developing an effective automated pipeline to curate these QA pairs is essential to address
the scarcity of large-scale, high-quality data for training and evaluating models in Olympiad-level
math reasoning. In this paper, we utilize the posts from the AoPS forum to create a large-scale
training and a contamination-resistant evaluation set. Our pipeline is designed to run automatically,
enabling us to build and maintain evolving train/evaluation datasets. This automated approach is
crucial, as it allows for continuously updating the datasets, ensuring they are less likely to suffer
from contamination, even as existing data potentially becomes compromised over time. In summary,
our key contributions are as follows:

• We build a pipeline to extract questions and solutions from raw AoPS forum data, con-
structing the AoPS-Instruct, a novel large-scale dataset with 666.1K Olympiad-level math
QA pairs.

• Using the most recent QA pairs, We build an automatic pipeline that introduces LiveAoPS-
Bench, a contamination-resistant evaluation set for assessing the math reasoning capabili-
ties of LLMs.

• Our experiments on LiveAoPSBench show a declining performance trend over time for
various LLMs, indicating potential data contamination, and stressing the need for up-to-
date evaluation data.

• Fine-tuning various LLMs on AoPS-Instruct lead to improved performance on standard
benchmarks such as OlympiadBench, Omni-Math, and our LiveAoPSBench dataset, veri-
fying the effectiveness of our dataset in enhancing math reasoning capabilities of LLMs.

2 RELATED WORK

In this section, we provide an overview of the existing mathematical datasets used for evaluation
and training purposes. Additionally, we review the latest methods and LLMs for enhancing and
evaluating these math datasets.

Evaluation Datasets for Math. The evaluation of the mathematical capabilities of LLMs has tra-
ditionally relied on well-established and widely-used datasets such as GSM8K and MATH (Cobbe
et al., 2021; Hendrycks et al., 2021b), which have served as benchmarks for several years. These
datasets typically contain math problems ranging from middle-school to high-school level, provid-
ing broad coverage across various problem categories. However, they present two significant lim-
itations: 1) being older, their test sets are more susceptible to contamination from current training
data of LLMs (Yang et al., 2023), and 2) they have reached a level of saturation, with state-of-the-art
(SOTA) models achieving over 90% accuracy (Yang et al., 2024b). To address these shortcom-
ings, Zhang et al. (2023b) introduced the Gaokao dataset, which includes more challenging high
school-level problems from the Chinese college entrance exam. In addition, newer datasets such as
OlympiadBench (He et al., 2024), AMC23 (AOPS, 2023), AIME24 (AOPS, 2024), and Omni-Math

1https://artofproblemsolving.com/community

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Model

2023
(5.2K)

2024
(3.8K)

Drop
(%)

7B-Qwen2.5-Math-Ins
72B-Qwen2-Math-Ins
72B-Qwen2.5-Math-Ins
72B-NuminaMath-CoT
20B-Internlm2-Math-Plus
16B-DeepSeekCoderV2-Ins
7B-Qwen2-Math-Ins
7B-NuminaMath-CoT
7B-DeepSeek-Math-RL
7B-Mathstral-v0.1
7B-Internlm2-Math-Plus

27B-Gemma2-it
70B-Llama-3.1-Ins
8B-Llama3.1-Ins
3B-Llama3.2-Ins
9B-Gemma2-it
1B-Llama-3.2-Ins

34.80
37.84
42.36
25.59
17.78
22.08
33.26
16.88
14.35
15.25
16.26

12.78
22.02
13.01
12.67
11.63
6.32

33.40
36.15
40.45
24.14
16.03
19.80
29.32
14.76
12.44
13.00
13.64

11.59
19.34
10.85
10.32
9.30
4.83

4.02
4.45
4.51
5.68
9.83
10.31
11.85
12.55
13.35
14.76
16.16

9.30
12.16
16.55
18.51
20.01
23.62

Figure 1: Accuracy trends of various LLMs on LiveAoPSBench over an 18-month period, high-
lighting a consistent decline in performance. We saperate math expert model with general purpose
model on the right. The degradation in accuracy varies across models, ranging from 2.4% to 23.6%.

(Gao et al., 2024) represent higher levels of difficulty, collecting from more recent high school com-
petition problems. While these datasets temporarily mitigate the risk of data contamination, they
remain susceptible to this issue as LLMs continue to evolve, particularly with fine-tuning on newer
data. To address this, we introduce LiveAoPSBench, which utilizes the most recent posts from the
AoPS forum and applies substring-matching techniques to exclude any previously used problems
from the new posts. More importantly, our pipeline is fully automated, allowing the evaluation set
to evolve with forum posts, thereby significantly decreasing the likelihood of contamination.

Training Datasets for Math. Training datasets for mathematical reasoning can be categorized into
two types: pretraining and supervised fine-tuning (SFT) datasets. First, pretraining datasets consist
of large-scale math data, e.g., billions of tokens used during the pretraining phase of LLMs. No-
table examples include Open-Web-Math (Paster et al., 2024) and Minerva (Lewkowycz et al., 2022),
which contain 38.5B and 14.7B tokens of math data, respectively. Second, SFT datasets focus on
high-quality question-answer pairs. Examples include Open-Math-Instruct (Toshniwal et al., 2024),
Orca-math (Mitra et al., 2024), and the training sets of widely used benchmarks such as GSM8K
(Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b). However, these datasets are generally
limited to grade-school level mathematics and do not target more advanced topics like Olympiad-
level math. One of the most closely related datasets to ours is Numina (Li et al., 2024), which com-
bines popular SFT datasets like Orca-math, MATH, and GSM8K, along with approximately 190K
new Olympiad-level QA pairs. Concurrently, Yue et al. (2024) introduced a large-scale instruc-
tion fine-tuning dataset for math and science, which has also shown improvements in mathematical
reasoning. Table 1 presents a detailed comparison of our dataset with these related datasets.

Contamination-Resistant Evaluation. Benchmarks that are publicly accessible are prone to be
contaminated due to the potential inadvertent data overlap during training. The typical decontam-
ination method involves using exact substring (e.g., n-gram) matching to detect overlaps with the
target evaluation sets (Zhuo et al., 2024). However, this approach fails to catch rephrased examples
and can not eliminate all overlaps with the test set.(Yang et al., 2023). While alternative LLM-based
methods for decontamination have been proposed, they often lack guarantees and may result in high
false-positive rates (Yang et al., 2023). A reliable way to mitigate contamination is to select data that
appeared after LLMs were trained, known as the knowledge cut-off. In the code generation domain,
LiveCodeBench (Jain et al., 2024) addresses this issue by categorizing data based on timestamps,
setting a cutoff date, and designating data beyond this point as unseen. We adopt a similar strategy
in the math domain, partitioning the dataset by timestamps and enabling users to select data based

3

Mar-May 23Jun-Aug 23Sep-Nov 23Dec-Feb 24Mar-May 24Jun-Aug 2451015202530354045Accuracy (%)Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Table 1: Comparison of our dataset with other related datasets from the literature. Our dataset
uniquely includes timestamp information and leverages open-source large language models (LLMs)s
like Qwen 2.5 72B for solution rewrites. ⋆ denotes inclusion of additional training datasets such as
GSM8K, Orca-Math, and MATH. Datasets marked with † have their solutions entirely generated by
LLM.

Dataset

Dataset Size
Train

Eval

Time Olympiad
Stamp

Level

Solution
Rewrite

Numina (Li et al., 2024)
OpenMathInstruct (Toshniwal et al., 2024)
OlympiadBench He et al. (2024)
GSM8K (Cobbe et al., 2021)
MATH (Hendrycks et al., 2021b)
Orca-Math (Mitra et al., 2024)

AoPS (Ours)

859K⋆
1.8M
−
7.5K
7.5K
200K

0.1K
−
6.1K
1K
5K
-

666.1K 3.9K

✗
✗
✗
✗
✗
✗

✓

✓
✗
✓
✗
✗
✗

✓

GPT4-o
Mixtral†
Human
Human
Human
GPT-4†

Qwen 2.5

on specific dates. Although this approach may not fully eliminate rephrased existing questions, it
ensures that evaluation data remains unseen and less contaminated, providing a more accurate and
fair assessment of LLMs.

Math-Specific Models. Several specialized models have been developed to improve the mathemat-
ical reasoning capabilities of LLMs (Shao et al., 2024; Mistral, 2024; Li et al., 2024; Yang et al.,
2024b; Azerbayev et al., 2024). These models are typically initialized from pretrained general-
purpose models, trained on large math datasets, followed by math-specific SFT, and then refined
through reinforcement learning with human feedback (RLHF). In this paper, we fine-tune both gen-
eral and math-specific models to demonstrate that AoPS-Instruct brings consistent improvements.

3 AOPS DATASET

In this section, we first describe the process of extracting and cleaning QA pairs from the AoPS
forum to construct our training set. Then we explain how to utilize the latest forum data to create a
reliable, contamination-resistant evaluation dataset for assessing model performance.

3.1 MATH INSTRUCTION FINE-TUNING DATASET: AOPS-INSTRUCT

We now describe the five steps of our automated pipeline for constructing the instruction fine-tuning
dataset AoPS-Instruct.

Step 0: Raw Forum Discussion Collection. We begin by collecting raw discussions from the forum
website, where each discussion is called a “topic”. In these topics, the author presents math problems
(e.g., competition-level problems) or general questions, such as seeking advice or resources. Our raw
dataset consists of 1, 076, 712 topics. Topics posted up until December 2023 are used as the training
set, while those posted between January and June 2024 are reserved as the evaluation dataset.

Step 1: Math Question Detection. We then filter out irrelevant topics, specifically those not con-
taining a mathematical question. To achieve this, we use Llama-3.1 8B (Dubey et al., 2024) to
decide the relevance of each topic. The first post of each topic determines whether the topic is a
mathematical question or not, so we manually design a few-shot prompt, provide the first post of the
topic to the model, and prompt the model to output if the topic is a math question or not. This step
reduces the dataset to 478, 337 topics with math questions after pruning 598, 375 irrelevant ones.

Step 2: Question-Answer Extraction. After filtering, we extract the math question from the first
post of each topic and identify potential solutions provided in subsequent posts. Since this task
requires understanding the entire conversation and determining which responses contain valid so-
lutions, we employ the 70B variant of Llama 3.1 for this step, enabling the detection of both the
question and all relevant answers from the discussion.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 2: The overall process of our dataset curation. Top: Dataset cleaning pipeline (Train-
ing). First, irrelevant topics are detected by a small LLM, then we extract questions and answers
from relevant discussions, and then each answer is rewritten into a step-by-step solution. Bottom:
LiveAoPSBench curation pipeline (Evaluation). We take the most recent posts, use two LLMs to
rewrite the solution, filter out the questions without clear final answer, and create the final evaluation
set.

Step 3: Solution rewriting. Math solutions generated by users on the AOPS forum are often con-
cise, omitting details assumed to be common knowledge among the target audience. For instance, a
xyz without explicitly mentioning the application of the AM-GM
user might write (x + yz)/2 ≥
inequality to (x, yz). While such brevity is typical for expert-level discussions, LLMs trained on
these succinct solutions often struggle to maintain their chain-of-thought reasoning capabilities.

√

Our experiments show that fine-tuning a model on these concise solutions significantly degrades
its performance on standard benchmarks (see Section 4.4 and Figure 5b). To address this issue,
we utilize the Qwen 2.5 72B model (Yang et al., 2024b) to rewrite all solutions into detailed, step-
by-step explanations. This approach aligns with similar techniques used in prior work, such as
the Numina project (Li et al., 2024), which also employed solution rewriting to improve response
quality. An example of a rewritten solution is provided in Figure 3, and the overall dataset curation
process is illustrated in Figure 2.

Step 4: Data Decontamination. After processing all the QA pairs, we apply data decontamina-
tion to remove any overlap with the test sets of commonly used math benchmarks. Following the
approach used in DeepSeekMath (Shao et al., 2024), we employ a 10-gram exact match decontami-
nation (Zhuo et al., 2024) method to ensure that our dataset remains distinct from those benchmarks.

After following the steps described above, we have a total of 675K QA pairs, out of which 666, 160
are before Jan 2024 and constitute the AoPS-Instruct. We provide further statistics of our dataset in
Section 4.1 and Figure 4.

3.2 CONTAMINATION-RESISTANT EVALUATION: LIVEAOPSBENCH

Math LLMs are trained on large instructional corpora. A common issue with current evaluation sets
is the risk of contamination, where test samples may inadvertently overlap with training data. To

5

Does anyone know ifmatrices or vectors areon the AMC tests?I don't rememberhaving seen it. If it's onthem, it's very rare.Compute  . is the samething as . Whichcanceling out termsgives us .Thank you!1.07M Topics478K TopicsTopicFilteringLlama8B478K TopicsCompute  . is the same thingas . Which canceling outterms gives us .Original A1QOriginal A2675K QA PairsCompute  . Step 2:   Step 3:  Step 4: Cancel out commonterms ... Final answer is Rewritten A2QAoPS-Instruct Pipeline LLM 1 Rewritten A1QLLM 2 Rewritten A1Original A1Repeat Answer Rewritingwith Multiple LLMTimestamp Cut-off Filter Out Proof QuestionsFilter Out No AnswerQBoxed AnsExtractionQualityFilterYesQ1Q2Quality VerificationBy HumanSubsampleLiveAoPSBench PipelineQAExtractionLlama70BAnswerRewritingQwen72BCompute  . is the samething as . Whichcanceling out termsgives us .Thank you! or  or ?Under review as a conference paper at ICLR 2025

Question: Find all solutions in N: 22x

+ 1 = 17y

User Response:
The multiplicative order of 2 modulo 17 is 8, since 24 = 16 ≡ −1 (mod 17). Therefore, if x ≥ 3,
then 8 | 2x, and so 22x

≡ 1 (mod 17). So x ≤ 2, and so the only solution is (x, y) = (2, 1).

+ 1 = 17y. If x ≥ 3, then 2x is a multiple of 8 (since 23 = 8 and higher

Qwen Rewritten Solution:
1. Determine the multiplicative order of 2 modulo 17. We find that 24 = 16 ≡ −1 (mod 17).
2. Since 24 ≡ −1 (mod 17), it follows that 28 ≡ 1 (mod 17). Therefore, the multiplicative order
of 2 modulo 17 is 8.
3. Consider the equation 22x
powers of 2 are multiples of 8).
4. If 2x is a multiple of 8, then 22x
5. Substituting 22x
to 2 ≡ 0 (mod 17). This is a contradiction.
6. Therefore, x must be less than 3. The possible values for x are 0, 1, and 2.
7. Check each possible value of x:
• For x = 0: 220
• For x = 1: 221
• For x = 2: 222

≡ 1 (mod 17) into the equation, we get 1+1 ≡ 17y (mod 17), which simplifies

+ 1 = 24 + 1 = 16 + 1 = 17, which is divisible by 17.

+ 1 = 21 + 1 = 3, which is not divisible by 17.

+ 1 = 22 + 1 = 5, which is not divisible by 17.

≡ 1 (mod 17), because the order of 2 modulo 17 is 8.

8. The only solution is (x, y) = (2, 1).
Thus, the solution is (2, 1) .

Figure 3: Example of a rewritten solution for a posted problem and a user solution. The model fills
in the trivial intermediate steps, reformats the response, and puts the final answer in a box.

create contamination-resistant benchmarks, we constructed our evaluation set by sorting the raw data
based on the initial posting timestamp and including only the most recent entries. Our evaluation set,
denoted as LiveAoPSBench, is sourced from the AoPS forum, with posts strictly between January
2023 and September 2024. We utilize the same pre-processing pipeline, depicted in Figure 2, to
extract QA pairs and have the raw solutions rewritten for consistency.

Filtering. The correctness of the solution is typically verified by comparing the final answer to the
human-annotated answer. Note that human-annotated answers may still contain errors, as we do
not perform formal proofs or verification. When constructing an evaluation set, it is essential that
each question has a concrete and definite answer, which is enclosed as ans format for ease of
parsing, as illustrated in Figure 3. We start by applying a series of heuristic filters to exclude proof-
based questions and extract only those with explicit, boxed answers. To ensure that our test set
does not contain problems included in widely used training sets, we use an stricter 8-gram matching
filter—stricter compared to the 10-gram filter used for training set decontamination. This helps
eliminate any potential overlap with common training corpora (Hendrycks et al., 2021b; Cobbe
et al., 2021; Mitra et al., 2024).

Cross-Check by LLMs. A key challenge in building a fair evaluation set is ensuring the accuracy
and validity of QA pairs. To automate this process, we employed two different models—Llama3.1-
70B-Ins (Dubey et al., 2024) and Qwen2-72B-Ins (Yang et al., 2024a) to perform the rewrit-
ing step twice for each question. Consequently, for each question Q, we obtain a triplet:
(Aqwen, Allama, Aoriginal). If a boxed answer is detected in Aoriginal, it is automatically accepted as
a candidate answer for the question. Following this, we performed a cross-check between Aqwen and
Allama, removing all cases with inconsistent answers. This was done through string matching for text
and value matching for numbers, while a SymPy-based (Meurer et al., 2017) symbolic equivalence
program was used for SymPy-parsable expressions. The final answers are obtained by deduplicat-
ing the candidate answers. Through this process, we constructed LiveAoPSBench, which contains
3,863 examples. Further details can be found in Appendix A.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Quality Verification. We assess the quality of our dataset by having a group of 10 graduate students
annotate a randomly selected 10% subset (386 cases) from our evaluation set. Each human annotator
verifies whether the final answer is correct based on the raw post, with each question annotated
by two different individuals. We report the percentage of cases marked as correct by the human
annotators to measure the correlation between human judgment and our method. Additionally, since
Olympiad-level questions can be challenging even for humans, we also report the inter-annotator
agreement to evaluate consistency between different groups of human annotators. More details can
be found in Section 4.4.

Evolving Evaluation with Up-to-date Data. Since our pipeline does not require human annotators,
we are able to continuously update our LiveAoPSBench in an automated manner. This makes our
benchmark an up-to-date and timestamped evaluation set that is resistant to contamination, thereby
providing a more reliable mathematical evaluation resource for the research community.

4 EXPERIMENTS

4.1 DATASET STATISTICS

We provide a better overview of the AoPS dataset in Figure 4. As shown in Figure 4a, more than
60% of the questions have only one answer, while around 24% and 8% have two and three answers,
respectively. Figure 4b shows the number of posts across each year, with a cut-off of August 2024.
We observe that each year at least 15K mathematical questions are posted to the forum. This trans-
lates to more than 1, 000 monthly questions, which shows the potential of the AoPS forum to be
used as training, and especially evaluation set. Figure 4c shows a breakdown of the types of ques-
tions in our dataset. Proof questions and numerical questions with about 32% and 28% constitute
the majority of the questions in our dataset.

Finally, Figure 4d shows the pairwise overlap between each pair of popular supervised fine-tuning
datasets using substring matching between the two datasets of each pair. Among the two Olympiad-
level datasets (i.e., ours and Numina), our dataset has the least overlap with common datasets (with
less than 14.1% overlap), which shows the number of new data points.

4.2 EVALUATING OPEN-SOURCED MODELS

We evaluate the models’ performance as a function of time window. As shown in Fig 1, we find
that all the models experience a performance drop when evaluating 2024 questions compared to
questions in 2023. This decline suggests that performance on earlier examples may not accurately
reflect the true capabilities of LLMs, as the initial results could be inflated by inadvertent data
overlap.

4.3

INSTRUCTION FINE-TUNING

We show that the collected training dataset is effective at improving the math reasoning capabilities
of LLMs. To this end, we choose 4 representative LLMs and fine-tune them on our dataset combined
with the Numina (Li et al., 2024) dataset, and show that such a combination provides superior
performance compared to training on either of the datasets alone.

We use the following set of diverse models for fine-tuning evaluation: (1) Mathstral-7B (Mistral,
2024): a math-specialized model derived from Mistral-7B (Jiang et al., 2023), (2) DeepSeekMath-
7B (Shao et al., 2024): a math-specialized model based on the DeepSeek family, and (3) Llama 3.2
3B (Dubey et al., 2024) and (4) Llama 3.2 1B (Dubey et al., 2024), two recent general state-of-the-
art models. For each QA pair, only the question is used as the instruction, with the rewritten solution
serving as the response, formatted within the model’s respective chat template. For instance, with
Mathstral, we use the prompt: <s>[INST] question [/INST]solution for instruction
tuning.

Consistent with prior work, we train each model for three epochs (Shao et al., 2024; Yang et al.,
2024b), as we observe additional epochs provide no further benefit (see Figure 9 in the Appendix
for ablation studies on the number of epochs). We explore three data mixtures for fine-tuning: (1)
AoPS alone, (2) Numina alone, and (3) AoPS + Numina. After fine-tuning each model, we eval-

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

(a) Number of answers per question.

(b) Number of questions per year, based on post date.

(c) Problem category distribution.

(d) Pairwise overlap between various datasets.

Figure 4: AoPS Dataset Statistics. The statistics are across all the datapoints in our dataset before
split. In (d), the percentage at row i and column j shows the fraction of the training set of i-th dataset
(based on exact substring match) present in the j-th dataset. Our dataset has the least overlap with
others with less than 14.1% overlap.

uate the performance of each model on the following standard competition-level benchmarks: (1)
OlympiadBench (He et al., 2024), which is an Olympiad-level evaluation dataset. Following prior
literature (Yang et al., 2024a), we take only the math questions which have final answers and do not
contain images or figures. This leaves us with 675 samples from this dataset (2) Omni-MATH (Gao
et al., 2024), which is a collection of 4428 problems from various mathematical olympiad competi-
tions. (3) LiveAoPSBench set for the year 2024. The results are shown in Table 2. As shown by the
table, fine-tuning with our dataset consistently boost the performance.

4.4 ABLATION STUDIES

Evaluation Quality Assessment. We assess the quality of our evaluation set in two ways: by mea-
suring its correlation with a well-established dataset and through manual evaluation over a subset
of the data. First, He et al. (2024) compiled an Olympiad-level math evaluation set using manual
assessment, which we leverage in our context to verify the quality of our method through the corre-
lation between accuracies. Figure 5a, demonstrates that the evaluation on LiveAoPSBench is highly
correlated with carefully established benchmarks such as OlympiadBench. This demonstrates that
our automatically generated benchmark aligns closely with the quality of those created through ex-
tensive human effort. Next, we subsample 10% of our evaluation set and ask human annotators to
verify the correctness of the final parsed answers by referring to the original post. Annotators are
given three options: yes, no, and no-answer. “Yes” and “no” indicate whether the answer is deemed
correct, while “no-answer” is selected when a concrete answer is not appropriate (e.g., abstract con-
cept questions answered with concrete examples). As a result, we found that 88% of the annotations
were marked as correct, while 8% were incorrect and 4% fell under the no-answer category. To
understand the gap from perfect accuracy here, we further measure the correlation between groups
of human annotators by computing the percentage of choices that were consistent. Surprisingly, the
human annotators only reached an agreement rate of 91%, demonstrating the challenge of evaluating
Olympiad-level problems, even for graduate-level annotators.

8

123456# Answers050000100000150000200000250000300000# Questions2003200420052006200720082009201020112012201320142015201620172018201920202021202220232024Year05000100001500020000250003000035000# PostsProof32.0%Numerical28.7%Expression17.7%Other11.6%Equation5.7%List4.3%MATHGSM8KORCA MathOpenMathIstNuminaAoPSMATHGSM8KORCA MathOpenMathIstNuminaAoPS1.0000.0000.0000.8700.8760.7270.0001.0000.9990.8670.9730.0000.0000.1621.0000.1410.8070.0010.7730.2230.2231.0000.8970.5660.0350.0420.1800.0671.0000.0740.0250.0000.0000.0220.0941.0000.00.20.40.60.81.0Under review as a conference paper at ICLR 2025

Table 2: Performance comparison of different models fine-tuned on various datasets across multiple
benchmarks. Bold values in the columns for No SFT, Numina, and AoPS-Ins represent the highest
scores for individual datasets. Additionally, bold values for Numina+AoPS-Ins indicate performance
that matches or surpasses all other fine-tuning alternatives. Our dataset outperforms Numina on most
benchmarks, and the combined (Numina+AoPS-Ins) fine-tuning consistently yields superior results.

Model

SFT Dataset

AoPS24 Math

Olympiad
Bench

Omni
Math

AIME24 AMC23

Deepseek-Math
7b-Ins

Mathstral
7B

Llama-3.2
3B-Ins

Llama-3.2
1B-Ins

No SFT
Numina
AoPS-Ins

Numina+AoPS-Ins

No SFT
Numina
AoPS-Ins

Numina+AoPS-Ins

No SFT
Numina
AoPS-Ins

Numina+AoPS-Ins

No SFT
Numina
AoPS-Ins

11.7
16.3
20.1

19.7

13.70
15.70
22.40

23.50

12.0
12.9
17.1

17.4

5.60
6.90
8.60

Numina+AoPS-Ins

10.50

47.1
55.5
62.3

58.8

56.30
54.60
60.30

60.60

47.4
49.5
52.9

55.6

28.80
32.70
34.70

36.60

14.5
22.7
22.4

25.6

21.20
23.40
23.40

27.30

16.1
19.3
18.5

22.8

4.70
6.40
11.10

10.40

12.3
17.0
18.3

18.0

15.90
17.10
17.60

20.10

12.9
14.4
15.1

17.2

7.00
9.70
11.00

10.30

1/30
0/30
0/30

2/30

0/30
0/30
1/30

2/30

2/30
1/30
0/30

0/30

0/30
0/30
0/30

0/30

8/40
12/40
16/40

11/40

16/40
15/40
14/40

14/40

11/40
6/40
11/40

12/40

5/40
6/40
6/40

6/40

(a) Correlation with the OlympiadBench dataset.

(b) Ablation on Rewriting.

Figure 5: Ablations on LiveAoPSBench.
(a) The performance of models on our benchmark is
highly correlated with established datasets. (b) The effect of rewriting user solutions into a step-by-
step solution with two different models. Rewriting solutions always improves accuracy, and using
stronger models leads to larger accuracy gains.

Rewritting’s effect on performance. We also ablate the effect of solution rewriting, which is an
important part of our pipeline. As shown in Figure 5b, rewriting solutions into a step-by-step format
substantially improves the test accuracy across all benchmarks. The Qwen-2.5 72B based rewriting
performs favorably against Llama-3.1 70b based rewriting on competition-level math benchmarks,
while being slightly worse on easier grade-school math. Overall, we found Qwen to be a stronger
model, providing a higher amount of details and being less verbose compared to Llama in rewriting
solutions (see Figure 17 in the Appendix for a qualitative example). This suggests that rewriting
solutions with stronger models can significantly improve performance on benchmarks.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

10152025303540Accuracy LiveAoPSBench (%)202530354045Accuracy Olympiad Bench (%)Qwen2.5-Math-72B-InsQwen2-Math-72B-InsQwen2-Math-1.5B-InsNuminaMath-7B-CoTMathstral-7B-v0.1DeepSeek-Coder-V2-Lite-InsGemma2-27b-itInternlm2-Math-plus-7bDeepSeek-Math-7b-rlQwen2-Math-7B-InsGemma2-9b-itQwen2.5-Math-7B-InsQwen2.5-Math-1.5B-InsSpearman Correlation: 0.9890MathGSM8KAMC23Olympiad BenchDatasets01020304050607080Accuracy (%)39.174.510.010.551.281.120.016.457.579.237.522.5Rewriting MethodRawLlama-3.1-70BQwen-2.5-72BUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

5 LIMITATIONS

Absence of Visual Content. Our dataset currently focuses on text-only problems, which may
limit its effectiveness in certain areas, particularly geometry. Many geometry problems rely heav-
ily on diagrams to fully convey the problem statement. Incorporating relevant images and figures
could significantly enhance the dataset’s comprehensiveness and applicability, especially in visually-
dependent mathematical domains.

Evaluation of Proof-based Questions. Our evaluation dataset focuses on QA pairs with clear, final
answers, which is well-suited to a broad range of Olympiad-level problems. However, a signifi-
cant portion of such types of problems involve more complex proof-based questions that require
detailed logical reasoning and multiple steps. While we incorporate proof-based questions in our
instruction-tuning pipeline, the current evaluation pipeline lacks the ability to evaluate such ques-
tions effectively.

Quality Variability in Community-Generated Content. The community-driven content from the
AoPS forum provides a rich source of high-quality data. Nevertheless, as with any community-
generated content, the quality of answers and solutions can vary. While our filtering and refinement
processes have successfully mitigated much of this noise, incorporating more advanced techniques
in future iterations could result in better consistency and precision.

6 CONCLUSION AND FUTURE WORK

In conclusion, this paper introduces the AoPS-Instruct dataset and LiveAoPSBench, leveraging
community-driven content from the Art of Problem-Solving forum to address the challenges of lim-
ited training data and unreliable evaluation for LLMs solving Olympiad-level math problems. By
developing a scalable and automated pipeline for extracting and refining question-answer pairs, this
work presents a dataset containing over 650, 000 QA pairs, along with an up-to-date, contamination-
resistant evaluation benchmark. Our experiments demonstrate significant performance improve-
ments across multiple standard benchmarks for models fine-tuned on the AoPS-Instruct, highlight-
ing enhanced mathematical reasoning capabilities. Furthermore, the observed performance decline
of various LLMs on LiveAoPSBench underscores the importance of continuously updating evalua-
tion sets to mitigate the risks of data contamination.

For future work, there are several promising directions to explore. First, while this paper focuses on
the AoPS forum, the pipeline developed is not limited to this domain. It is generalizable and can
be applied to other online forums or different subject areas, enabling the creation of high-quality
datasets for various fields, such as physics, computer science, or even non-technical disciplines. Ex-
panding this pipeline to other knowledge-intensive communities could further improve the training
and evaluation of LLM across disciplines. Additionally, the quality of the dataset can be signifi-
cantly improved by incorporating more advanced LLMs into the pipeline. Leveraging state-of-the-
art models for question extraction, answer detection, and solution rewriting would result in more
accurate and detailed data, ultimately enhancing the effectiveness of the fine-tuned models. Lastly,
the current pipeline focuses on question-answer pairs with clear final answers, but a significant por-
tion of Olympiad-level problems involves proof-based questions that require a deeper evaluation of
logical reasoning, argument structure, and intermediate steps. Future work could include adapting
the pipeline to accommodate these proof-based problems, potentially using another advanced LLM
as a judge (Li et al., 2023), or incorporating formalization methods to better assess these complex
solutions.

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

AIMO. The aimo prize. https://aimoprize.org, November 2023.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

AOPS. 2023 amc 12a, and 12b problems. https://artofproblemsolving.com/wiki/
index.php/2023_AMC_12A_Problems, https://artofproblemsolving.com/
wiki/index.php/2023_AMC_12B_Problems, 2023.

AOPS. 2024 aime community page. https://artofproblemsolving.com/community/

c3370201_2024_aime, 2024.

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer,
Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model
for mathematics. In The Twelfth International Conference on Learning Representations, 2024.
URL https://openreview.net/forum?id=4WnqRR915j.

S´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka-
mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general
intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John
Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,
2021.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
arXiv preprint arXiv:2407.21783x qx q, 2024.

Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma,
Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark
for large language models. arXiv preprint arXiv:2410.07985, 2024.

Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han,
Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench:
A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scien-
tific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the
62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa-
pers), pp. 3828–3850, Bangkok, Thailand, August 2024. Association for Computational Linguis-
tics. doi: 10.18653/v1/2024.acl-long.211. URL https://aclanthology.org/2024.
acl-long.211.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob
Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna-
tional Conference on Learning Representations (ICLR), 2021a.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS,
2021b.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando
Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free
evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot,
Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.

Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi.
Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north
american chapter of the association for computational linguistics: human language technologies,
pp. 1152–1157, 2016.

Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski,
Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo,
Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative rea-
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave,
soning problems with language models.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openreview.net/forum?id=IFXTZERXdM7.

Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi
Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin
Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu.
Numina-
math. https://github.com/project-numina/aimo-progress-prize/blob/
main/report/numina_dataset.pdf, 2024.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following
models. https://github.com/tatsu-lab/alpaca_eval, 5 2023.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R´emi Leblond, Tom
Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation
with alphacode. Science, 378(6624):1092–1097, 2022.

Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondˇrej ˇCert´ık, Sergey B. Kirpichev,
Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath-
nayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam
Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, ˇStˇep´an Rouˇcka,
Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy:
symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992.
doi: 10.7717/peerj-cs.103. URL https://doi.org/10.7717/peerj-cs.103.

Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing
english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pp. 975–984, 2020.

Mistral. Mathstral blog. https://mistral.ai/news/mathstral/, 2024.

Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking

the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024.

Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open
dataset of high-quality mathematical web text. In The Twelfth International Conference on Learn-
ing Representations, 2024. URL https://openreview.net/forum?id=jKHmjlpViu.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple
math word problems? In Proceedings of the 2021 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language Technologies, pp. 2080–
2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.
naacl-main.168. URL https://aclanthology.org/2021.naacl-main.168.

Daniel Selsam, Leonardo de Moura, Kevin Buzzard, Reid Barton, Percy Liang, Sarah Loos, and
Freek Wiedijk. Imo grand challenge. https://imo-grand-challenge.github.io/,
2019.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li,
Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open
language models. arXiv preprint arXiv:2402.03300, 2024.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu-
patiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, et al. Gemma
2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.

Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git-
arXiv preprint

man. Openmathinstruct-1: A 1.8 million math instruction tuning dataset.
arXiv:2402.10176, 2024.

Trieu Trinh, Yuhuai Wu, Quoc Le, He He, and Thang Luong. Solving olympiad geometry without

human demonstrations. Nature, 2024. doi: 10.1038/s41586-023-06747-5.

12

Under review as a conference paper at ICLR 2025

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
arXiv preprint

Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.
arXiv:2407.10671, 2024a.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian-
hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical
expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024b.

Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. Rethinking
benchmark and contamination for language models with rephrased samples. arXiv preprint
arXiv:2311.04850, 2023.

Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma,
Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models
toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the

web. arXiv preprint arXiv:2405.03548, 2024.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma-
chine really finish your sentence? In Annual Meeting of the Association for Computational Lin-
guistics, 2019. URL https://api.semanticscholar.org/CorpusID:159041722.

Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen.
Evaluating and improving tool-augmented computation-intensive math reasoning. arXiv preprint
arXiv:2306.02408, 2023a.

Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the

performance of large language models on gaokao benchmark. 2023b.

Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li,
Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models
in code intelligence. arXiv preprint arXiv:2406.11931, 2024.

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari,
Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench-
marking code generation with diverse function calls and complex instructions. arXiv preprint
arXiv:2406.15877, 2024.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A MORE DETAILS ON LIVEAOPSBENCH

A.1 EVALUATION PIPELINE STATISTICS

To begin with, we have 14158 QA pairs with time stamps between Jan-2024 and Aug-2024. Decon-
tamination with 8-gram matching is performed against Math and GSM8K training set (Hendrycks
et al., 2021b; Cobbe et al., 2021), which removes 664 Q-A pairs. After removing proof questions
and non-boxed solutions, we are left with 7173 Q-A pairs over 5416 unique questions. Lastly, The
LLM cross-check filters out 1553 questions with inconsistent solutions and the resulting LiveAoPS-
Bench contains 3863 questions. We apply the same pipeline described in Sec 3.2 to data with a time
stamp between Jan-2023 and Dec-2023 and get 5216 questions for the 2023 split result.

A.2 HUMAN ANNOTATION

As shown in Figure 6, we develop a simple web interface for human annotators to verify the answers
extracted by our LLMs. Annotators compare the “Voted Answer”, “Original Answers” and all posts
in the original topic page identified by LLMs to verify if the “Voted Answer” matches the original
posts’ answers. The verification process provides four results: Positive (“Yes”), negative (“No/No
Answer”), and neutral (“Not sure”). The “Not sure” option is provided since verifying the answer
sometimes requires a certain mathematical foundation and a significant amount of reading time. We
also show highlight two examples of disagreement in Figure 7.

A.3 DERIVATION OF DIFFICULTY LEVELS

The difficulty levels in this dataset do not reflect the exact difficulty of the problems but rather ap-
proximate the general education background of the problem, e.g., this is a “High School” level prob-
lem. However, a challenging high school problem may be more complex than an easy college-level
problem. The classification is derived from the problem tag in the AOPS forum, where the cate-
gories correspond to “Middle School”, “High School”, “College”, and “High School Olympiads”.
In addition, some problems originate from special forums, which do not fit into the above categories
and are classified as “Others” in our dataset.

Figure 6: Human Annotation Interface.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Example 1

Question: In a triangle, ABC, Angle BAC = 90◦; AD is the altitude from A onto BC. Draw DE
perpendicular to AC and DF perpendicular to AB. Suppose AB = 15 and BC = 25. Then the
length of EF is?

Raw Post:
Because the triangle is a right triangle, so by the Pythagorean Theorem, the length of AC is 25² - 15²
= 20. The area of ABC is AB * AC / 2 = 15 * 20 / 2. But it can also be represented by 25 * AD /
2. Putting them together we get 15 * 20 / 2 = 25 * AD / 2. So AD = 15 * 20 / 25 = 12. Because DE
perpendicular to AC and DF perpendicular to AB, AEDF is a rectangle, which means that EF = AD,
EF = 12. The answer is A.

Voted Answer: 12

Is the Voted answer consistent with answer in raw post?
Human Annotator 1: ✓
Human Annotator 2: ✗

Example 2

Question: For a positive integer k we write (1 + x)(1 + 2x)(1 + 3x)......(1 + kx) = a0 + a1x +
a2x2 + ....... + akxk where a0, ...ak are the coefficients of the polynomial. Find the sum of all the
digits of smallest possible value of k if a0 + a1 + a2 + ......a(k − 1) is divisible by 2005.

Raw Post:
f (x) = (1 + x)(1 + 2x) . . . (1 + kx) = a0 + a1x + . . . + akxk
a0 + a1 + . . . + ak−1 = f (1) − ak
ak = 1 · 2 · 3 . . . k = k!
f (1) = 2 · 3 · 4 . . . (1 + k) = (k + 1)!
2005 | (k + 1)! − k! =⇒ 2005 | k · k!
2005 = 5 · 401
k ≥ 401

Voted Answer: 5

Is the Voted answer consistent with answer in raw post?
Human Annotator 1: ✗
Human Annotator 2: ✓

Figure 7: We highlight two examples of annotation inconsistencies caused by human annotators:
1. Example 1: Annotator 2 failed to recognize that the answer is explicitly stated in the raw post.
2. Example 2: The raw post does not directly provide the final answer. Annotator 1 was unable to
reason that 4 + 0 + 1 = 5 constitutes the correct solution.

B DETAILED EVALUATION RESULTS ON LIVEAOPSBENCH

B.1 EVALUATING OPEN-SOURCED LLMS

We have selected several mainstream open-source general LLMs and math-specific LLMs that
demonstrate high performance on the previous math evaluation datasets. For math-specific LLMs,
we choose DeepSeek-Math-7b-rl (Shao et al., 2024), Mathstral-7B-v0.1 (Mistral, 2024), 7b and 20b
versions of Internlm2-Math-plus (Ying et al., 2024), 7B and 72B versions of NuminaMath-CoT (Li
et al., 2024), 1.5B,7B,72B version of Qwen2-Math-Instruct (Yang et al., 2024a) and Qwen2.5-Math-
Instruct (Yang et al., 2024b) as the representative of the math specific LLMs. Additionally, we in-
clude DeepSeek-Coder-V2-Lite-Instruct (Zhu et al., 2024), which is a code specialist model trained
on both math and code corpus. For general purpose LLMs, We report performance on 1B, 3B and

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Model

Count

DeepSeek-Coder-V2-Lite-it
DeepSeek-Math-7b-rl
Internlm2-Math-plus-20b
Internlm2-Math-plus-7b
Mathstral-7B-v0.1
NuminaMath-72B-CoT
NuminaMath-7B-CoT
Qwen2-Math-1.5B-it
Qwen2-Math-72B-it
Qwen2-Math-7B-it
Qwen2.5-Math-1.5B-it
Qwen2.5-Math-72B-it
Qwen2.5-Math-7B-it

Llama-3.2-1B-it
Llama-3.2-3B-it
Llama-3.1-8B-it
Gemma-2-27b-it
Gemma-2-9b-it

2023

5216

22.45
15.38
18.23
17.10
15.91
26.15
17.29
29.06
37.96
33.07
34.72
42.04
34.87

6.75
13.77
14.03
13.80
12.42

Jan

483

23.40
18.22
18.84
17.81
17.18
29.40
18.43
31.47
41.41
33.13
36.02
44.31
35.82

6.83
14.49
15.53
11.39
10.97

Model

Count

DeepSeek-Coder-V2-Lite-it
DeepSeek-Math-7b-rl
Internlm2-Math-plus-20b
Internlm2-Math-plus-7b
Mathstral-7B-v0.1
NuminaMath-72B-CoT
NuminaMath-7B-CoT
Qwen2-Math-1.5B-it
Qwen2-Math-72B-it
Qwen2-Math-7B-it
Qwen2.5-Math-1.5B-it
Qwen2.5-Math-72B-it
Qwen2.5-Math-7B-it

Llama-3.2-1B-it
Llama-3.2-3B-it
Llama-3.1-8B-it
Gemma-2-27b-it
Gemma-2-9b-it

Feb

388

19.33
15.46
16.75
16.49
14.95
25.77
14.95
26.55
36.34
35.31
32.47
38.40
31.70

5.93
13.40
13.66
13.92
10.05

2024

3863

20.86
13.64
16.93
14.81
14.29
24.95
16.13
26.84
36.68
30.05
32.57
40.56
34.04

5.80
11.75
12.71
12.87
10.67

Mar

444

22.97
17.57
17.79
19.82
15.54
23.87
18.24
31.31
37.39
33.33
35.81
41.22
36.26

4.73
12.16
14.86
15.77
13.51

Jan

634

18.14
14.04
14.51
11.83
11.04
22.08
13.56
25.24
34.38
28.08
28.71
40.54
32.18

3.63
8.68
10.57
10.57
8.68

Apr May

415

472

22.17
13.49
18.31
17.11
14.22
25.30
15.90
31.08
38.31
31.33
34.94
41.93
35.42

5.54
13.98
13.73
13.01
12.53

Feb

527

22.77
17.08
22.01
18.60
17.65
28.65
20.87
29.79
37.57
31.69
33.59
41.56
35.48

7.40
13.85
16.13
16.13
11.95

22.67
14.41
20.55
18.01
16.31
29.87
19.70
28.81
38.35
36.86
37.50
45.55
37.71

9.53
14.19
13.56
15.04
13.35

Mar

614

25.41
15.31
17.59
17.59
17.43
28.66
17.10
30.46
41.69
34.20
39.25
45.11
41.37

6.84
12.87
14.17
13.03
13.36

Jun

412

21.12
12.86
17.72
15.05
16.75
22.57
15.53
26.21
37.14
28.64
32.04
44.17
29.61

5.10
13.11
11.65
10.68
11.17

Jul

396

23.99
15.66
16.41
19.95
16.67
25.51
17.68
26.77
37.37
35.10
31.31
40.15
33.59

8.08
16.92
15.15
14.90
13.38

Aug

505

25.74
14.65
18.42
16.24
18.22
25.54
17.43
29.90
38.61
33.27
36.04
42.18
37.43

6.73
12.08
16.04
15.64
15.05

Sep

381

22.83
15.75
17.06
13.91
13.91
24.41
15.22
27.82
34.91
30.45
33.86
35.43
32.28

8.66
14.44
12.86
12.86
10.76

Oct

409

23.72
15.89
21.27
18.58
14.18
27.87
18.83
29.58
40.10
32.52
36.67
46.21
34.72

7.09
15.16
14.91
14.91
13.45

Nov

404

22.28
17.08
18.81
17.08
18.32
25.50
19.80
32.43
39.60
36.39
35.15
43.32
37.87

7.92
13.61
14.60
14.36
13.61

Dec

507

18.93
13.61
16.57
15.19
14.20
27.02
15.38
26.43
35.50
30.57
33.73
40.43
34.52

5.13
12.43
11.64
13.02
10.85

Apr May

503

511

22.47
15.71
17.50
16.50
16.50
23.46
19.68
27.24
36.98
31.41
31.21
40.16
32.41

6.16
14.12
11.33
13.52
10.93

18.00
10.37
14.48
11.15
10.96
23.48
13.89
22.90
36.59
27.98
31.70
37.96
32.09

4.70
10.18
12.33
11.94
8.02

Jun

380

17.89
10.53
14.74
17.11
12.11
22.37
13.42
25.79
35.79
29.74
30.00
40.53
32.11

5.53
13.42
12.63
11.58
8.16

Jul

363

21.21
13.50
18.73
13.77
12.95
27.55
16.25
28.37
36.91
30.85
34.99
42.42
34.99

7.16
9.92
12.67
14.05
14.05

Aug

331

19.64
9.97
15.71
10.88
15.11
22.36
12.69
23.56
30.82
24.17
29.61
33.23
28.40

5.44
11.18
11.48
12.39
10.27

Table 3: Accuracy per Month for Different Models

8B versions of the Llama3 family models (Dubey et al., 2024) as well as 9B and 27B versions of
Gemma-2-Instruct (Team et al., 2024) model.

B.2 DETAILED RESULTS

The accuracy comparison for these mainstream open source LLMs are shown in Tables 3, 4, 5 split
by Month, Difficulty and Answer Type. The Month tables separately include evaluation results for
2023 and 2024. For the Difficulty and Answer Type tables, we use only the most recent evaluation
results from 2024. Notably, the difficulty labels represent the general educational background of
the problems rather than their exact difficulty. Over half of the problems originate from educational
backgrounds associated with High School or High School Olympiads, and only around 7% are from
Middle School, indicating our dataset’s focus is more on the complex problems. Similarly, in the
Answer Type Table, more than half of the problems are categorized as numeric-int.

C TRAINING SET DETAILS

C.1 DECONTAMINATION DETAILS

We use 10-gram substring matching to decontaminate against test set for a comprehensive list of
math evaluation datasets available. (Cobbe et al., 2021; Hendrycks et al., 2021b; He et al., 2024;
AOPS, 2023; 2024; Zhang et al., 2023b; Lewkowycz et al., 2022; Gao et al., 2024; Miao et al., 2020;
Hendrycks et al., 2021a; Koncel-Kedziorski et al., 2016; Patel et al., 2021; Zhang et al., 2023a). In
Figure 8. We show the decontamination statistic for our dataset and Numina.

16

Under review as a conference paper at ICLR 2025

Table 4: Accuracy per Difficulty for Different Models: The difficulty labels are for general education
background of the problem and do not reflect the exact difficulty of the problem.

Model

Count

DeepSeek-Coder-V2-Lite-it
DeepSeek-Math-7b-rl
Internlm2-Math-plus-20b
Internlm2-Math-plus-7b
Mathstral-7B-v0.1
NuminaMath-72B-CoT
NuminaMath-7B-CoT
Qwen2-Math-1.5B-it
Qwen2-Math-72B-it
Qwen2-Math-7B-it
Qwen2.5-Math-1.5B-it
Qwen2.5-Math-72B-it
Qwen2.5-Math-7B-it

Llama-3.2-1B-it
Llama-3.2-3B-it
Llama-3.1-8B-it
Gemma-2-27b-it
Gemma-2-9b-it

Overall Middle School High School College High School Others

Olympiads

3863

20.86
13.64
16.93
14.81
14.29
24.95
16.13
26.84
36.68
30.05
32.57
40.56
34.04

5.80
11.75
12.71
12.87
10.67

286

24.48
22.73
24.83
20.63
19.23
33.57
18.18
32.17
43.36
37.06
38.81
48.25
42.66

10.14
18.53
17.48
20.63
15.38

1349

19.79
12.23
15.64
13.94
12.90
25.28
15.12
26.17
38.62
30.17
33.65
42.70
34.84

4.30
10.23
10.60
11.86
8.30

314

22.93
14.97
19.11
17.20
15.61
26.75
16.24
28.98
42.04
31.21
31.85
45.86
36.62

5.41
9.55
15.29
13.38
12.74

889

17.21
8.55
12.71
9.67
10.24
18.45
12.60
22.61
28.01
24.86
28.91
32.62
27.11

3.71
8.10
7.54
7.65
7.65

1025

23.80
16.98
19.41
18.05
17.85
27.22
19.90
29.27
38.15
32.10
32.78
40.88
35.80

8.49
15.71
17.85
16.39
14.44

Table 5: Accuracy per Answer Type for Different Models: As not all answers can be easily ver-
ified, we divide the answers into different types to facilitate more accurate comparison and more
convenient observation of the structural distribution of the dataset.

list

195

20.00
10.77
13.33
12.31
6.67
16.92
11.28
28.72
36.41
27.69
32.82
41.54
33.85

2.56
7.18
10.77
9.74
8.72

numeric-dec

numeric-int

numeric-irr

others

57

15.79
8.77
14.04
12.28
12.28
24.56
12.28
15.79
26.32
19.30
22.81
31.58
24.56

5.26
7.02
17.54
14.04
14.04

2114

24.36
16.18
20.20
18.31
16.65
28.71
18.78
29.52
41.15
33.30
35.86
43.19
36.90

7.66
14.71
15.33
15.28
12.54

176

11.93
6.25
8.52
8.52
10.80
20.45
12.50
25.57
31.82
25.00
22.73
36.36
30.11

3.41
6.25
9.66
9.09
7.95

75

21.33
21.33
18.67
22.67
21.33
32.00
21.33
32.00
40.00
37.33
46.67
45.33
40.00

5.33
16.00
21.33
16.00
16.00

Model

Count

DeepSeek-Coder-V2-Lite-it
DeepSeek-Math-7b-rl
Internlm2-Math-plus-20b
Internlm2-Math-plus-7b
Mathstral-7B-v0.1
NuminaMath-72B-CoT
NuminaMath-7B-CoT
Qwen2-Math-1.5B-it
Qwen2-Math-72B-it
Qwen2-Math-7B-it
Qwen2.5-Math-1.5B-it
Qwen2.5-Math-72B-it
Qwen2.5-Math-7B-it

Llama-3.2-1B-it
Llama-3.2-3B-it
Llama-3.1-8B-it
Gemma-2-27b-it
Gemma-2-9b-it

Overall

equation

expression

3863

20.86
13.64
16.93
14.81
14.29
24.95
16.13
26.84
36.68
30.05
32.57
40.56
34.04

5.80
11.75
12.71
12.87
10.67

296

18.24
11.15
14.53
11.49
14.53
19.59
14.86
23.31
27.70
23.99
25.34
31.42
28.72

2.70
7.77
4.05
7.77
7.09

950

16.00
10.42
12.74
9.26
10.74
20.21
12.11
22.11
30.84
26.21
28.74
38.32
30.21

3.79
8.32
9.58
10.11
7.89

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 8: Decontamination Statistics: We perform decontamination on the raw dataset to produce
AoPS-Instruct, with the same method as the Numina-Math-COT. Both datasets show considerable
overlap with the MATH dataset. AoPS-Instruct exhibits more contamination within our 2024 split
due to repeated questions, while Numina-Math-COT has higher contamination with other external
datasets, reflecting its multi-source composition.

Figure 9: Ablation study on accuracy with respect to training steps. Here, 18,000 steps approxi-
mately correspond to 6 epochs for AoPS-Instruct and Numina, and 3 epochs for AoPS-Instruct +
Numina. We can see that LiveAoPSBench + Numina consistenly improve as training goes on.

D SFT EXPERIEMENTS

D.1 ABLATION WITH CONTROLLED COMPUTATION BUDGET

As shown in Tab 2, Numina + AoPS-Instruct performs favorably against using AoPS-Instruct or
Numina alone. To show this gain is not simply achieved by doubling the computation avail-
able for fine-tuning. We control the total fine-tune budget the same for AoPS-Instruct only, nu-
mina only and AoPS-Instruct + numina. This results in approximately 6 epoch on AoPS-Instruct
or Numina or 3 epoch of training on AoPS-Instruct + Numina. We show the curve of ACC on
Math,LiveAoPSBench, OlympiadBench w.r.t. training steps.

D.2 REWRITING MODEL ABLATION

We use Qwen 2.5 72B to rewrite the solutions, and then we fine-tune smaller models on our dataset.
This may raise the question of whether the effectiveness of our dataset would be limited by the
capabilities of its rewriting model. To show the effectiveness of our dataset, we use a Qwen 2.5
1.5B to rewrite the solutions and then fine-tune DeepSeek-Math 7B-instruct on the dataset. Table

18

math_questionsasdivmmlu_stemomni_mathmawpssvampaops_2024olympiad_benchcarp_enOthersExternal Test Sets100010000Number of Matches (Exact/10 Gram)2201809343676001464921160599154575396503246804590181417171251635344Ours-Raw (Total: 43992)NuminaMath-CoT (Total: 40916)600080001000012000140001600018000Training Steps12131415161718AccuracyLiveAoPSBenchModelsNuminaAoPS-InsAoPS-Ins + Numina600080001000012000140001600018000Training Steps4850525456AccuracyMathModelsNuminaAoPS-InsAoPS-Ins + Numina600080001000012000140001600018000Training Steps151617181920212223AccuracyOlympiadBenchModelsNuminaAoPS-InsAoPS-Ins + NuminaUnder review as a conference paper at ICLR 2025

Table 6: Performance comparison of original DeepSeek-Math, Qwen2.5-1.5B, and DeepSeek-Math
fine-tuned on solutions rewritten by Qwen2.5-1.5B. The fine-tuned DeepSeek-Math significantly
outperforms both the original model and the rewriting model, demonstrating that our dataset en-
hances reasoning capabilities beyond the limitations of its rewriting model.

Model

AIME24 AMC23 Olympiad Bench Math AoPS24 Omni Math

Deekseek-Math-7b-Ins
Qwen2.5-1.5b-Ins
Deepseek-Math-7b-Ins (fine-tuned)

1/30
0/30
1/30

8/40
9/40
13/40

14.5
21.3
22.7

47.1
55.0
61.0

11.7
16.7
19.4

12.3
16.8
19.2

Insturction:
You are given an online Math QA post. Your task is to identify whether the post asked is a concrete
mathematical question, note that this means it shouldn’t be an abstract general question related to math,
and output the result as \boxed{0} for no and \boxed{1} for yes. A few examples are provided below:

Few shot examples...

Now, your task is to provide output for the following post:

Post: Post 1

Example Classify result:

\boxed{0/1}

Figure 10: Prompt for the Topic Filtering part in Fig 2.

6 shows the performance of the original DeepSeek-Math, the performance of Qwen2.5-1.5B, and
the performance of fine-tuned DeepSeek on Qwen2.5-1B-rewritten solutions. As shown by the
results, the fine-tuned version outperforms both models, which shows that our dataset can improve
the reasoning capability beyond its rewriting model solution.

E USE OF AOPS AS A DATA SOURCE

Concurrent to our work, both Numina (Li et al., 2024) and Omni-math (Gao et al., 2024) also use
AoPS as their data source. Different from us, Numina only includes data from the contest page
with 30K questions 2, while we utilize all the 1.07 available posts on this forum. Furthermore,
Omni-math (Gao et al., 2024) includes only 4428 evaluation questions from all timestamps, while
we include the most recent problems posted in 2024, as well as a large-scale training set.

F PROMPTS

We provide the Prompts used in our pipeline in Figures 10, 11, and 12.

G DATASET EXAMPLES

We provide further examples of our dataset and its rewritten solutions in Figures 13, 14, 15, 16. 17,
and 18.

2https://artofproblemsolving.com/community/c13_contest_collections

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Insturction:
You are given an online Math QA forum where each user post in each topic is in the format ”post i by
user j: [post i text]”. Each user may reply to other users by quoting their post. Your task is to identify
the question asked by the first user and find all potential answers within the follow-up discussion, and
output them in a structured json format.
Your output json must have a ”question” key containing the question, and one ”answers” key contain-
ing the list of answers. Each answer must have three keys: a ”user” key to identify the user who posted
the solution, a ”post number” to identify which post number the answer originates from, and a ”con-
tent” key for the content of the solution. Make sure to reformat the answer content to make it a formal
clean solution, without missing details. Do not include any irrelevant information in the answer. Do
not add any additional information to the question or answers.
Ensure to handle different line breaks and spaces between posts accurately, and maintain the sequence
of the dialogue. Always surround mathematical questions with $ symbols for LaTeX formatting. In
case the dialogou does not contain any mathematical question, or there are no valid answers, leave the
”question” or ”answer” key empty. A few examples are proided below:

Few shot examples...

Now, your task is to provide JSON output for the following Topic:

post 1 by user1: ...
post 2 by user2: ...

Example Parse result:
{

”question”: ”Question from Post 1”,
”answers”: [
{

”user”: ”User2”,
”post number”: 2,
”content”: ”Solution 1”

},
{

}”

]

}

”user”: ”User4”,
”post number”: 5,
”content”: ”Solution 2”

Figure 11: Prompt for the QA extraction part in Fig 2.

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Insturction:
You are given a solution to a mathematical question. Your task is to re-write the solution into a
step-by-step solution with itemized steps(1..., 2...., 3....). You should re-write the solution in a
formal and clean way, without missing any details. Make sure to include all the necessary steps and
explanations. Do not include any irrelevant information in the answer. Do not add any additional
information to the solution. Always surround mathematical questions with $ symbols for LaTeX
formatting. If the question contains a final number, make sure to put the number in a

boxed for LaTeX formatting.

Question: Question

Solution: Raw Answer

Now provide the formalized re-written answer. DO NOT include the question in your answer.
Only respond with the re-written solution.

Example Rewrite result:
Rewritten solutions

Figure 12: Prompt for the Solution Rewriting Part in Fig 2.

21

Under review as a conference paper at ICLR 2025

Question: Find all pairs (k, l) of positive integers such that kl

lk = k!

l!

User Response 1:
W.l.o.g we may assume k ≥ l.
If k ≥ l + 2, then k − 1 divides the RHS, thus the LHS, thus kl, but k, k − 1 are coprime, thus this is
only possible if k = 2, which would imply l = 0, contradiction.
So the cases k = l, l + 1 are left.
If k = l, then we always have equality.
If k = l + 1, then the we need kk−1
(k−1)k = k ⇐⇒ kk−2 = (k − 1)k.
Again k − 1 is supposed to divide a power of k, thus we remain with k = 2, so with l = 1, clearly
being a solution.

User Response 2: If (k,l) solution, then (l,k) is solution too. Therefore we can find solutions k ≥ l. If
l=1 we have k = k! =⇒ k = 1 or k = 2.
If l=2 we have k2 = 2k−1k! solution is k=2.
If l ≥ 3 then kl
lk ≤ 1 and k!
All solutions (k, l) = (1, 2)(2, 1), (n, n), n ∈ N .

l! ≥ 1, therefore solution is l=k.

Qwen Rewritten Solution 1:
1. Without loss of generality, assume k ≥ l.
2. Consider the case k ≥ l + 2:

• The right-hand side (RHS) of the equation kl
• Therefore, the left-hand side (LHS) must also be divisible by k − 1.
• Since k and k − 1 are coprime, k − 1 must divide kl.
• This is only possible if k = 2, which would imply l = 0, leading to a contradiction.

l! is divisible by k − 1.

lk = k!

3. Therefore, the only possible cases are k = l and k = l + 1.
4. If k = l:

• The equation kl

lk = k!

l! simplifies to kk

kk = k!

k! , which is always true.

5. If k = l + 1:

ll+1 = (l+1)!

• The equation becomes (l+1)l
• Simplifying, we get (l+1)l
• This further simplifies to (l + 1)l−1 = ll+1.
• Again, l and l + 1 are coprime, so l must divide a power of l + 1.

ll+1 = l + 1.

l!

.

• The only solution is l = 1, which implies k = 2.

6. Therefore, the pairs (k, l) that satisfy the equation are (k, l) = (k, k) and (k, l) = (2, 1) .

If (k, l) is a solution, then (l, k) is also a solution. Therefore, we can assume without loss of

Qwen Rewritten Solution 2:
1.
generality that k ≥ l.
2. Consider the case when l = 1. The equation becomes k1
1k = k!
only positive integer solutions to this equation are k = 1 and k = 2.
3. Consider the case when l = 2. The equation becomes k2
2k = k!
The only positive integer solution to this equation is k = 2.
4. Consider the case when l ≥ 3. We have kl
lk ≤ 1 and k!
simultaneously, it must be that k = l.
5. Therefore, the solutions are (k, l) = (1, 2), (2, 1), and (n, n) for any positive integer n.

l! ≥ 1. For these two inequalities to hold

2! . This simplifies to k2 = 2k−1k!.

1! , which simplifies to k = k!. The

Figure 13: An example illustrating two distinct rewritten solutions for a single posted problem,
where each user has proposed a unique approach to solving it.

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Question: Let ABC be an equilateral triangle. On the plane (ABC) rise the perpendiculars AA′ and
BB′ on the same side of the plane, so that AA′ = AB and BB′ = 1
2 AB. Determine the measure the
angle between the planes (ABC) and (A′B′C ′).

User Response:
Denote A′B′ ∩ AB = D.Then BD = BA = BC so ∠ACD = 90◦. Thus the plane AA′C is
perpendicular to CD and the angle between A′B′C and ABC is equal to ∠A′CA = 45◦.

Qwen Rewritten Solution:
1. Denote the intersection of A′B′ and AB as D.
2. Since BD = BA = BC, it follows that △BDA and △BDC are isosceles triangles.
3. Therefore, ∠ACD = 90◦.
4. The plane AA′C is perpendicular to CD.
5. The angle between the planes (A′B′C ′) and (ABC) is equal to ∠A′CA.
6. Since AA′ = AB and BB′ = 1
triangle, ∠A′CA = 45◦.
7. Thus, the measure of the angle between the planes (ABC) and (A′B′C ′) is 45◦ .

2 AB, and given the symmetry and properties of the equilateral

Figure 14: An example of a rewritten geometry solution for a posted problem.

23

Under review as a conference paper at ICLR 2025

Question: Solve the equation in parameter p: x3 + px2 + (p − 1 + 1

p−1 )x + 1 = 0

User Response: Note that p ̸= 1. Then x = 1 − p works, you can get the other roots (when they
exist) by quadratic formula.

Qwen Rewritten Solution:
1. First, observe that the parameter p must satisfy p ̸= 1 to avoid division by zero in the term
2. Next, consider the given equation:

(cid:16) 1
p−1

(cid:17)
.

x3 + px2 +

(cid:18)

p − 1 +

(cid:19)

1
p − 1

x + 1 = 0

3. We test the value x = 1 − p to see if it is a root of the equation. Substitute x = 1 − p into the
equation:

(1 − p)3 + p(1 − p)2 +

4. Simplify each term:

(cid:18)

p − 1 +

1
p − 1

(cid:19)

(1 − p) + 1 = 0

(1 − p)3 = 1 − 3p + 3p2 − p3
p(1 − p)2 = p(1 − 2p + p2) = p − 2p2 + p3

(cid:18)

p − 1 +

(cid:19)

1
p − 1

(1−p) = (p−1)(1−p)+

1 − p
p − 1

5. Combine all the simplified terms:

= (p−1)(1−p)−1 = −p2+2p−1−1 = −p2+2p−2

1 − 3p + 3p2 − p3 + p − 2p2 + p3 − p2 + 2p − 2 + 1 = 0

6. Simplify the expression:

(1 − 3p + 3p2 − p3) + (p − 2p2 + p3) + (−p2 + 2p − 2) + 1 = 0

1 − 3p + 3p2 − p3 + p − 2p2 + p3 − p2 + 2p − 2 + 1 = 0
1 − 3p + 3p2 − p3 + p − 2p2 + p3 − p2 + 2p − 2 + 1 = 0
0 = 0

7. Since the equation holds true, x = 1 − p is indeed a root of the equation.
8. To find the other roots, we can perform polynomial division of the original cubic polynomial by
(x − (1 − p)) to obtain a quadratic polynomial.
9. The quadratic polynomial can then be solved using the quadratic formula:

−b ±

x =

√

b2 − 4ac
2a

where a, b, and c are the coefficients of the quadratic polynomial obtained from the division.
10. The final roots of the equation are:

x = 1 − p and the roots of the quadratic polynomial obtained from the division.

11. The solution to the equation is:

x = 1 − p

Figure 15: An example of an incomplete solution where the author left finding the other two roots
of the equation to the reader. While the language model has adequately and correctly expanded the
solution of the author, two more solutions of the equation are not mentioned (neither by the author
5−6p+p2
nor the LLM): − p−1+

5−6p+p2

and

1−p+

√

√

.

2(p−1)

2(p−1)

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Question: Find a 5 × 5 checkerboard with a square removed that cannot be tiled using right triomi-
noes(L shaped). Prove that such a tiling does not exist for this board.

User Response:
Alright, then colour
01010
11111
01010
11111
01010
The sum of the squares covered by each L is at least 2, so if we place 8 L’s we get a total of at least 16,
meaning that every 1 must be covered. Hence removing a 1 makes the covering impossible.

Qwen Rewritten Solution:
1. Consider a 5 × 5 checkerboard with the following coloring pattern:

0
1
0
1
0

1
1
1
1
1

0
1
0
1
0

1
1
1
1
1

0
1
0
1
0

2. Each right triomino (L-shaped) covers exactly 3 squares. When placed on the board, the sum of the
numbers on the squares covered by each L-shaped triomino is at least 2. This is because each L-shaped
triomino must cover at least one square with a 1 and two squares with 0s or 1s.
3. If we place 8 L-shaped triominoes on the board, the total sum of the numbers on the squares covered
by these 8 triominoes is at least 8 × 2 = 16.
4. The total number of squares on the 5 × 5 board is 25. Removing one square leaves 24 squares. Since
each L-shaped triomino covers 3 squares, 8 L-shaped triominoes will cover exactly 24 squares.
5. For the 8 L-shaped triominoes to cover the 24 squares, the sum of the numbers on these 24 squares
must be at least 16. Given the coloring pattern, the sum of the numbers on all 25 squares is 13 (since
there are 12 squares with 0 and 13 squares with 1).
6. If a square with a 1 is removed, the sum of the numbers on the remaining 24 squares is 13 − 1 = 12.
7. Since the sum of the numbers on the squares covered by the 8 L-shaped triominoes must be at least
16, and the sum of the numbers on the remaining 24 squares is only 12, it is impossible to cover the
board with 8 L-shaped triominoes if a square with a 1 is removed.
8. Therefore, a 5 × 5 checkerboard with a square removed that cannot be tiled using right triominoes
(L-shaped) is one where a square with a 1 is removed. The tiling is impossible in this case.
1

Figure 16: A failure example for rewriting the solution. While the Qwen model gets the overall
logic of the proof correct, it produces mistakes in counting the number of “ones” on the checkboard
and provides incorrect intermediate steps.

25

Under review as a conference paper at ICLR 2025

Question: Let x be a real number such that log2 x + log4 5 + log8 7x = log√
User Response:
Turn everything into log2 of something. Here is the algebra:

2 x5/6. Find x.

log2 x + log22 5 + log23 7x = log21/2 x5/6,
x5,
log2 x + log2
5 + log2
5 · 3√
√

3√
3√
7x = log2
7x) = log2( 3√

√

log2(x ·
5 · 3√

7x = 3√

√

x ·

x5 =⇒ x = 35

x5),
√

5 .

Qwen Rewritten Solution:
1. Convert all logarithms to base 2: log2 x + log22 5 + log23 7x = log21/2 x5/6
2. Simplify the logarithms using the change of base formula:
1
1
2

log2 x + 1

log2 x5/6

2 log2 5 + 1

3 log2 7x =

(cid:16)

= log2

√
5 · 3√

2 log2 5 + 1

3 log2 7x = 2 log2 x5/6

x · 51/2 · (7x)1/3(cid:17)
(cid:16)

3. Simplify the right-hand side: log2 x + 1
4. Use the property of logarithms to combine the terms on the left-hand side: log2 x + log2 51/2 +
log2 (7x)1/3 = log2 x5/3
x5/3(cid:17)
5. Combine the logarithms on the left-hand side: log2
6. Since the logarithms are equal, the arguments must be equal: x · 51/2 · (7x)1/3 = x5/3
7. Simplify the equation: x ·
8. Divide both sides by x (assuming x ̸= 0):
9. Raise both sides to the power of 3 to eliminate the cube root: (
10. Simplify the equation: 53/2 · 7x = x2
11. Simplify 53/2: 53/2 = 5
12. Substitute back into the equation: 5
5x = x2
13. Simplify the equation: 35
14. Rearrange the equation to form a quadratic equation: x2 − 35
15. Factor out x: x(x − 35
5) = 0
16. Solve for x: x = 0 or x = 35
17. Since x = 0 is not a valid solution (as it would make the original logarithms undefined), the

7x)3 = (x2/3)3

7x = x5/3
√

5 · 7x = x2

7x = x2/3

5 · 3√

5 · 3√

5x = 0

√

√

√

√

√

√

√

5

5

solution is: x = 35

5

√

Llama Rewritten Solution:
To solve the given equation, we first need to express all logarithms in terms of log2. This can be
achieved by applying the change of base formula for logarithms.
Step 1: Express log4 5 and log8 7x in terms of log2.
Using the change of base formula, we have: log4 5 = log2 5
log2 23 = log2 7x
log8 7x = log2 7x
2 x5/6 in terms of log2.
Step 2: Express log√

log2 22 = log2 5

log2 8 = log2 7x

log2 4 = log2 5

2 = log2

3 = log2

5 and

7x.

3√

√

2

√

x5

3√

3√

log2

= log2 x5/6

7x = log2
5 · 3√

2 x5/6 = log2 x5/6
x5.

log2 21/2 = log2 x5/6
√

Using the change of base formula, we have: log√
3√
2 log2 x5/6 = log2 (x5/6)2 = log2 x5/3 = log2
Step 3: Substitute the expressions from Steps 1 and 2 into the original equation. log2 x + log2
log2
Step 4: Combine the logarithms on the left-hand side using the product rule for logarithms.
log2 (x ·
Step 5: Since the logarithms are equal, the arguments of the logarithms must be equal. Therefore, we
have:
7x = 3√
√
5 · 3√
x ·
Step 6: Solve for x.
7x = 3√
5 · 3√
√

x5 =⇒ x = 35

7x) = log2

5 +

x5.

x ·

3√

x5

√

√

=

1
2

5 .

Figure 17: Comparison of Qwen 2.5 72B and Llama 3.1 70B on rewriting solutions. Overall, the
Qwen model shows a higher level of detail in expanding equations compared to Llama.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

Question: Solve the equation in parameter p: x3 + px2 + (p − 1 + 1

p−1 )x + 1 = 0

(cid:16)

(cid:17)

(cid:16)

(cid:17)

(cid:17)

(cid:16)

p−1

x + 1

p−1 = 0

p − 1 + 1
p−1

p − 1 + 1
p−1

x + 1 = 0 in the

· 1 + 1 f (1) = 1 + p + p − 1 + 1

p−1 + 1 f (1) = 2p + 1 + 1

p−1 = 0 Multiply through by p − 1 to clear the fraction:

Qwen Generated Solution: To solve the equation x3 + px2 +
parameter p, we will follow a systematic approach. Let’s denote the polynomial by f (x):
f (x) = x3 + px2 +
### Step 1: Check for Rational Roots We will use the Rational Root Theorem to check for possible
rational roots. The Rational Root Theorem states that any rational root, expressed in its lowest terms
p
q , must have p as a factor of the constant term (1) and q as a factor of the leading coefficient (1).
Therefore, the possible rational roots are ±1.
### Step 2: Test x = 1 Substitute x = 1 into the polynomial:
f (1) = 13 + p · 12 +
p − 1 + 1
p−1
For x = 1 to be a root, f (1) = 0:
2p + 1 + 1
### Step 3: Solve the Equation 2p + 1 + 1
(2p + 1)(p − 1) + 1 = 0 2p2 − 2p + p − 1 + 1 = 0 2p2 − p = 0
Factor out p:
p(2p − 1) = 0
This gives us two solutions:
p = 0 or p = 1
2
### Step 4: Verify the Solutions We need to verify if these values of p indeed make x = 1 a root of
the original polynomial.
#### For p = 0: Substitute p = 0 into the polynomial:
f (x) = x3 + 0 · x2 +
Check f (1):
f (1) = 13 − 2 · 1 + 1 = 1 − 2 + 1 = 0
So, x = 1 is a root when p = 0.
#### For p = 1
f (x) = x3 + 1
x3 + 1
2 x2 − 3
Check f (1):
2 · 12 − 3
f (1) = 13 + 1
2 · 1 + 1 f (1) = 1 + 1
So, x = 1 is not a root when p = 1
2 .
### Conclusion The only value of p for which x = 1 is a root of the polynomial is p = 0.
Thus, the solution to the equation is:
0

2 : Substitute p = 1
(cid:16) 1
2 x2 +

2 − 1 − 2(cid:1) x + 1 f (x) =

2 + 1 f (1) = 1 + 1 − 1 = 1

x + 1 f (x) = x3 − 2x + 1

x + 1 f (x) = x3 + 1

2 into the polynomial:

2 − 1 + 1
1
2 −1

0 − 1 + 1
0−1

2 x2 + (cid:0) 1

2 x + 1

2 − 3

(cid:16)

(cid:17)

(cid:17)

Figure 18: An example of letting Qwen generate the entire solution without the user input (from
Figure 15). In this case, the model completely fails to respond correctly, misinterpreting the question
and taking an incorrect approach to solving the problem. This shows the cruciality of rewriting
solutions (rather than generating entire solutions from stronger models).

27

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

