|
--- |
|
license: llama2 |
|
--- |
|
|
|
We have released HSPMATH-7B, a supervised fine-tuning model for MATH. |
|
|
|
We constructed a supervised fine-tuning dataset of 75k samples through a simple yet effective method based on the MetaMathQA dataset. |
|
After supervised fine-tuning the Llemma-7B model, we achieved a strong performance of 64.3% on the GSM8K dataset. |
|
The dataset construction method involves introducing a hint before the solution. For details, refer to the paper: [Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge](https://arxiv.org/pdf/2402.14310.pdf). |
|
|
|
|
|
A comparison of performances with methods of similar model sizes (7B) is shown in the table below: |
|
|
|
|
|
| Open-source Model (7B) | GSM8k | |
|
|-----------|------------| |
|
|MetaMath-Mistral-7B|77.7 | |
|
|MetaMath-7B-V1.0| 66.5 | |
|
|HSPMATH-7B| **64.3** | |
|
|Llemma-7B (SFT)| 58.7 | |
|
|WizardMath-7B| 54.9 | |
|
|RFT-7B |50.3| |
|
|Qwen-7b|47.84 |
|
|Mistral-7b|37.83 | |
|
|Yi-6b| 32.6 | |
|
|ChatGLM-6B| 32.4 | |
|
|LLaMA2-7b|12.96 | |
|
|
|
|Close-source Model|GSM8k| |
|
|-----------|------------| |
|
|GPT-3.5 | 57.1 | |
|
|PaLM-540B |56.5 | |
|
|Minerva-540B |58.8 | |
|
|Minerva-62B |52.4 | |
|
|Chinchilla-70B |43.7| |
|
|
|
Note: |
|
- The MetaMath family models is fine-tuned on 400k samples, which is more than 5.3 times the size of our training set. |
|
- Llemma-7B (SFT) and our model HSPMATH-7B are supervised fine-tuning (SFT) on the same dataset but without the Hint texts. |
|
- We found that by introducing hints, the SFT model HSPMATH-7B improved by 5.6%. |
