|
--- |
|
base_model: |
|
- marcuscedricridia/Hush-Qwen2.5-7B-della1 |
|
- marcuscedricridia/Hush-Qwen2.5-7B-della2 |
|
- marcuscedricridia/Hush-Qwen2.5-7B-RP-1M |
|
- marcuscedricridia/Hush-Qwen2.5-7B-della3 |
|
- marcuscedricridia/Hush-Qwen2.5-7B-della4 |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
|
|
--- |
|
## Performance Highlights |
|
|
|
Hush-Qwen2.5-7B-Preview was created using the YoYo v3 merge technique, achieving a new high on the IFEVAL test for 7B models with a score of **79.62%**. This makes it the **second-best** model in that category, though the leading model is currently unavailable, meaning we might be in **first place** by default! |
|
|
|
### Strengths |
|
- **High IFEVAL Score:** 79.62%, among the best for 7B models. |
|
- **Well-rounded performance:** Decent scores across various benchmarks. |
|
|
|
### Weaknesses |
|
- **Low MATH Score:** 35%, which is significantly lower than our past models (which scored at least 45%). Improving this would make the model substantially better overall. |
|
|
|
## Benchmark Results |
|
| Category | Score (%) | |
|
|-----------|----------| |
|
| Average | 35.13 | |
|
| IFEVAL | 79.62 | |
|
| BBH | 35.33 | |
|
| MATH | 37.54 | |
|
| GPQA | 8.17 | |
|
| MUSR | 12.73 | |
|
| MMLU | 37.38 | |
|
|
|
## Next Steps |
|
|
|
- **Finetune on Math:** Bringing up the math score is a priority to create a well-balanced model. |
|
- **Explore YoYo v4:** The next step could be merging this model with another one that is strong in math using the YoYo v4 technique. However, YoYo v4 lacks proper documentation, making it a challenge to implement. |
|
- **Develop a Math-Strong Model:** An alternative approach is to build a new model that performs decently in all benchmarks but excels in math, then merge it with this one. |
|
|
|
## Conclusion |
|
Hush-Qwen2.5-7B-Preview is a strong contender in the IFEVAL category, achieving one of the highest scores among 7B models. However, improving the math benchmark is a key priority for future iterations. By either finetuning or leveraging new merge techniques like YoYo v4, we can push the model to new heights. |
|
|
