|
--- |
|
license: llama3.1 |
|
datasets: |
|
- survivi/Llama-3-SynE-Dataset |
|
- hfl/stem_zh_instruction |
|
- llamafactory/alpaca_zh |
|
- llamafactory/alpaca_gpt4_zh |
|
- hfl/ruozhiba_gpt4 |
|
- codingsteven/Llama-3-8B-chat |
|
language: |
|
- zh |
|
metrics: |
|
- accuracy |
|
base_model: |
|
- meta-llama/Llama-3.1-8B |
|
model-index: |
|
- name: Control-LLM-Llama3.1-8B-SynE-Full-Parameter-Tuning |
|
results: |
|
- task: |
|
type: pretraining-evaluation |
|
dataset: |
|
type: mixed |
|
name: Pretraining Evaluation Dataset |
|
metrics: |
|
- name: exact_match,strict-match (meta_pretrain) |
|
type: exact_match |
|
value: 0.45445720757159036 |
|
stderr: 0.0035036029889520047 |
|
verified: false |
|
- name: exact_match,strict-match (meta_bbh_3shot_cot_pretrain) |
|
type: exact_match |
|
value: 0.6482875134387959 |
|
stderr: 0.005918167158231359 |
|
verified: false |
|
- name: acc,none (meta_mmlu_5shot_pretrain) |
|
type: accuracy |
|
value: 0.649480131035465 |
|
stderr: 0.004026616190778244 |
|
verified: false |
|
- name: exact_match,strict-match (meta_mmlu_pro_5shot_pretrain) |
|
type: exact_match |
|
value: 0.34956781914893614 |
|
stderr: 0.004347262544061378 |
|
verified: false |
|
- task: |
|
type: chinese-evaluation |
|
dataset: |
|
type: mixed |
|
name: Chinese Evaluation Dataset |
|
metrics: |
|
- name: acc,none (ceval-valid) |
|
type: accuracy |
|
value: 0.5898959881129272 |
|
stderr: 0.012699457390113113 |
|
verified: false |
|
- name: exact_match,strict-match (ceval-valid-pretrain-cot_zh) |
|
type: exact_match |
|
value: 0.40193164933135217 |
|
stderr: 0.01265090064840271 |
|
verified: false |
|
- name: acc,none (cmmlu) |
|
type: accuracy |
|
value: 0.6018822310481782 |
|
stderr: 0.004420298073040671 |
|
verified: false |
|
- name: exact_match,strict-match (cmmlu_pretrain_cot_zh) |
|
type: exact_match |
|
value: 0.4425833189431877 |
|
stderr: 0.004506238417180843 |
|
verified: false |
|
pipeline_tag: text-generation |
|
library_name: transformers |
|
--- |
|
|
|
# Control-LLM-Llama3.1-8B-SynE-Full-Parameter-Tuning |
|
This is a fine-tuned model of Llama-3.1-8B for muliligual-Chinese tasks on SynE dataset. |
|
|
|
## Linked Paper |
|
This model is associated with the paper: [Control LLM: Controlled Evolution for Intelligence Retention in LLM](https://huggingface.co/papers/2501.10979). |
|
|
|
## Linked Open Source code - training, eval and benchmark |
|
This model is associated with the github: [Control-LLM](https://github.com/linkedin/ControlLLM). |
|
|
|
## Evaluation Results |
|
Here is an overview of the evaluation results and findings: |
|
|
|
### Benchmark Results Table |
|
|
|
The table below summarizes evaluation results across Chinese tasks and original capabilities. |
|
|
|
| **Model** | **CEval** | **CEvalC** | **CMMLU** | **CMMLUC** | **C-Avg** | **BBH** | **MLU** | **MLUP** | **O-Avg** | **Overall** | |
|
|--------------------|-----------|------------|-----------|------------|-----------|---------|---------|----------|-----------|-------------| |
|
| Llama3.1-8B | 48.3 | 12.8 | 51.1 | 14.1 | 13.9 | 65.2 | 65.4 | 35.5 | 45.9 | 29.9 | |
|
| Llama-3-SynE | 57.7 | 22.3 | 57.1 | 22.8 | 22.8 | 61.9 | 64.0 | 32.6 | 42.9 | 32.9 | |
|
| **Full Param Tune**| 59.0 | 40.2 | **60.2** | 44.3 | 43.8 | 64.8 | 64.9 | 35.0 | 45.4 | 44.6 | |
|
| Stack Expansion | 56.0 | 32.7 | 55.2 | 33.4 | 33.3 | 62.3 | 65.6 | 35.3 | 44.8 | 39.1 | |
|
| Concat-Lerp* | 57.1 | 34.8 | 57.0 | 37.4 | 37.1 | 64.4 | 64.6 | 35.8 | 45.9 | 41.5 | |
|
| **Hybrid Expansion**| **58.9** | 44.7 | 57.9 | 44.3 | 44.4 | 65.1 | **65.7**| 36.9 | 46.8 | 45.6 | |
|
| **Control LLM*** | 57.0 | **44.7** | 56.0 | **44.9** | **44.8** | **68.2**| 65.6 | **37.9** | **48.5** | **46.7** | |
|
|
|
--- |
|
|
|
### Explanation: |
|
- **CEval**: Chinese Evaluation |
|
- **CEvalC**: Chinese Evaluation (CoT - Chain of Thought) |
|
- **CMMLU**: Chinese MMLU |
|
- **CMMLUC**: Chinese MMLU (CoT) |
|
- **C-Avg**: Chinese - Size Weighted Average across CEval, CEvalC, CMMLU, and CMMLUC |
|
- **BBH**: BigBench Hard |
|
- **MLU**: MMLU (Massive Multitask Language Understanding) |
|
- **MLUP**: MMLU Pro |
|
- **O-Avg**: Original Capability - Size Weighted Average across BBH, MLU, and MLUP |
|
- **Overall**: Combined average across all tasks |
|
|
|
### Full Parameter Tuning on Chinese-SynE |
|
The following plot illustrates the Catastrophic Forgetting of full parameter tuning in terms of hidden states alignment drift. |
|
|
|
 |
