|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
WIP |
|
|
|
|
|
This is an early experiment on laser and its influence on language understanding. |
|
|
|
Will keep you updated.. |
|
|
|
### German benchmarks |
|
|
|
| **German tasks:** | **MMLU-DE** | **Hellaswag-DE** | **ARC-DE** |**Average** | |
|
|-------------------------------|-------------|---------------|--------------|--------------| |
|
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | | |
|
| _7B parameters_ | | | | | |
|
| llama-2-7b | 0.400 | 0.513 | 0.381 | 0.431 | |
|
| leo-hessianai-7b | 0.400 | 0.609 | 0.429 | 0.479 | |
|
| bloom-6b4-clp-german | 0.274 | 0.550 | 0.351 | 0.392 | |
|
| mistral-7b | **0.524** | 0.588 | 0.473 | 0.528 | |
|
| leo-mistral-hessianai-7b | 0.481 | 0.663 | 0.485 | 0.543 | |
|
| leo-mistral-hessianai-7b-chat | 0.458 | 0.617 | 0.465 | 0.513 | |
|
| DPOpenHermes-7B-v2 | 0.517 | 0.603 | 0.515 | 0.545 | |
|
| hermeo-7b | 0.511 | **0.668** | **0.528** | **0.569** | |
|
| **germeo-7b-laser (this model)**| ? | ? | ? | ? | |
|
| _13B parameters_ | | | | | |
|
| llama-2-13b | 0.469 | 0.581 | 0.468 | 0.506 | |
|
| leo-hessianai-13b | **0.486** | **0.658** | **0.509** | **0.551** | |
|
| _70B parameters_ | | | | | |
|
| llama-2-70b | 0.597 | 0.674 | 0.561 | 0.611 | |
|
| leo-hessianai-70b | **0.653** | **0.721** | **0.600** | **0.658** | |
|
|
