README.md

---
library_name: transformers
license: mit
datasets:
- OLAIR/Open-R1-Ko-SFT-v2.0
language:
- ko
base_model:
- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
---

# Model Card: OLAIR/ko-r1-7b-v2.0.3

This document describes the OLAIR/ko-r1-7b-v2.0.3 model, including its training data, intended use, performance benchmarks, limitations, and ethical considerations.

---

## 1. Overview

**Model Name:** OLAIR/ko-r1-7b-v2.0.3  
**Model Type:** Large Language Model (LLM) for Korean language understanding and reasoning  
**Version:** 2.0.3

This model is designed to provide Korean language capabilities with a focus on reasoning tasks. It is the second version in its series, building upon previous iterations with improvements in training data and fine-tuning methodologies.


## 2. Training Data

The model was trained on the dataset provided by OLAIR, specifically the [Open-R1-Ko-SFT-v2.0](https://huggingface.co/datasets/OLAIR/Open-R1-Ko-SFT-v2.0) dataset. This dataset includes a curated collection of Korean language data, optimized for supervised fine-tuning (SFT) to enhance reasoning and natural language understanding capabilities in Korean.


## 3. Benchmark Performance

The model's performance has been evaluated using the HAE-RAE Reasoning Challenge (HRC), which measures reasoning abilities across various domains. Below are the benchmark results for several models, including OLAIR/ko-r1-7b-v2.0.3:
We've noticed some errors in the previous code and updated it.

| Model                                 | Chemistry | Math   | Physics | Physics Word Puzzles | Puzzles | Average |
|---------------------------------------|-----------|--------|---------|----------------------|---------|---------|
| o1-2024-12-17                         | 57.14     | 78.18  | 77.78   | 80.00                | 84.62   | 75.54   |
| o3-mini-high                          | 57.14     | 81.82  | 77.78   | 70.00                | 69.23   | 71.19   |
| o3-mini-2025-01-31                     | 50.00     | 80.00  | 70.37   | 50.00                | 76.92   | 65.46   |
| o1-mini-2024-09-12                     | 42.86     | 56.36  | 70.37   | 60.00                | 15.38   | 48.99   |
| Deepseek-R1                           | 50.00     | 54.55  | 62.96   | 70.00                | 7.69    | 49.04   |
| gpt-4o-2024-11-20                      | 35.71     | 32.73  | 51.85   | 50.00                | 53.85   | 44.83   |
| Exaone-3.5-32B-Instruct               | 21.43     | 30.91  | 25.93   | 50.00                | 38.46   | 33.35   |
| Qwen2.5-72B-Instruct                  | 35.71     | 30.91  | 51.85   | 20.00                | 23.08   | 32.31   |
| **Ko-R1-7B-v2.0.3**                   | 7.14      | 61.82  | 40.74   | 40.00                | 0.00    | 29.94   |
| Ko-R1-7B-v1                           | 7.14      | 63.64  | 37.04   | 40.00                | 0.00    | 29.56   |
| gpt-4o-mini-2024-07-18                 | 21.43     | 29.09  | 37.04   | 50.00                | 0.00    | 27.51   |
| UNIVA-Bllossom_DeepSeek-llama3.1-Bllossom-8B | 28.57     | 16.36  | 33.33   | 10.00                | 15.38   | 20.73   |


<details>
<summary>Depricated Score</summary>  
  
| Model                                 | Chemistry | Math  | Physics | Physics Word Puzzles | Puzzles | Average |
|---------------------------------------|-----------|-------|---------|----------------------|---------|---------|
| o1-2024-12-17                         | 42.9      | 74.5  | 77.8    | 70.0                 | 30.8    | 59.2    |
| o3-mini-high                          | 35.7      | 72.7  | 70.4    | 70.0                 | 23.1    | 54.4    |
| o3-mini-2025-01-31                     | 35.7      | 74.5  | 74.1    | 60.0                 | 7.7     | 50.4    |
| o1-mini-2024-09-12                     | 35.7      | 54.5  | 63.0    | 60.0                 | 0.0     | 42.6    |
| Deepseek-R1                           | 35.7      | 52.7  | 51.9    | 60.0                 | 0.0     | 40.1    |
| gpt-4o-2024-11-20                      | 28.6      | 21.8  | 37.0    | 50.0                 | 0.0     | 27.5    |
| **Ko-R1-7B-v2.0.3**                   | **7.1**   | **56.4**  | **29.6**    | **40.0**                 | **0.0**     | **26.6**    |
| Qwen2.5-72B-Instruct                  | 35.7      | 29.1  | 37.0    | 30.0                 | 0.0     | 26.4    |
| Ko-R1-7B-v1                           | 0.0       | 60.0  | 22.2    | 40.0                 | 0.0     | 24.4    |
| Exaone-3.5-32B-Instruct               | 28.6      | 27.3  | 22.2    | 40.0                 | 0.0     | 23.6    |
| gpt-4o-mini-2024-07-18                 | 7.1       | 29.1  | 22.2    | 50.0                 | 0.0     | 21.7    |
| UNIVA-Bllossom_DeepSeek-llama3.1-Bllossom-8B | 14.3      | 10.9  | 33.3    | 0.0                  | 0.0     | 11.7    |
  
</details> 




*Note:* The above table reflects performance across multiple reasoning domains. The metrics indicate that while OLAIR/ko-r1-7b-v2.0.3 shows competitive performance in certain areas (e.g., Math), there remain challenges, particularly in Chemistry and Physics-related tasks, compared to some higher-performing counterparts.


## 4. Limitations

- The model is still vulnerable to Korean-related inputs, leading to endless loops of thinking. We are working to fix it.

## ETC
How to Cite

```
To be added
```

Contact 
```
[email protected]
```