File size: 1,717 Bytes
45444a8 b7577f8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
---
license: apache-2.0
---
WIP
This is an early experiment on laser and its influence on language understanding.
Will keep you updated..
### German benchmarks
| **German tasks:** | **MMLU-DE** | **Hellaswag-DE** | **ARC-DE** |**Average** |
|-------------------------------|-------------|---------------|--------------|--------------|
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | |
| _7B parameters_ | | | | |
| llama-2-7b | 0.400 | 0.513 | 0.381 | 0.431 |
| leo-hessianai-7b | 0.400 | 0.609 | 0.429 | 0.479 |
| bloom-6b4-clp-german | 0.274 | 0.550 | 0.351 | 0.392 |
| mistral-7b | **0.524** | 0.588 | 0.473 | 0.528 |
| leo-mistral-hessianai-7b | 0.481 | 0.663 | 0.485 | 0.543 |
| leo-mistral-hessianai-7b-chat | 0.458 | 0.617 | 0.465 | 0.513 |
| DPOpenHermes-7B-v2 | 0.517 | 0.603 | 0.515 | 0.545 |
| hermeo-7b | 0.511 | **0.668** | **0.528** | **0.569** |
| **germeo-7b-laser (this model)**| ? | ? | ? | ? |
| _13B parameters_ | | | | |
| llama-2-13b | 0.469 | 0.581 | 0.468 | 0.506 |
| leo-hessianai-13b | **0.486** | **0.658** | **0.509** | **0.551** |
| _70B parameters_ | | | | |
| llama-2-70b | 0.597 | 0.674 | 0.561 | 0.611 |
| leo-hessianai-70b | **0.653** | **0.721** | **0.600** | **0.658** |
|