Hugging Face's logo

Intel
/
opt-13b-int4-inc

English
Model card Files Community
yintongl commited on
Commit
f48f2b0
·
verified ·
1 Parent(s): 2c9725f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -48,17 +48,17 @@ Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.gi
48
  lm_eval --model hf --model_args pretrained="Intel/opt-13b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu --batch_size 32
49
  ```
50
 
51
- | Metric | FP16 | INT4 |
52
  | -------------- | ------ | ------ |
53
- | Avg. | 0.4988 | 0.5021 |
54
- | mmlu | 0.2455 | 0.2456 |
55
  | lambada_openai | 0.6858 | 0.6949 |
56
  | hellaswag | 0.5247 | 0.5177 |
57
- | winogrande | 0.6472 | 0.6448 |
58
  | piqa | 0.7590 | 0.7573 |
59
  | truthfulqa_mc1 | 0.1971 | 0.2056 |
60
  | openbookqa | 0.2680 | 0.2780 |
61
- | boolq | 0.6587 | 0.6801 |
62
  | arc_easy | 0.6713 | 0.6717 |
63
  | arc_challenge | 0.3294 | 0.3251 |
64
 
 
48
  lm_eval --model hf --model_args pretrained="Intel/opt-13b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu --batch_size 32
49
  ```
50
 
51
+ | Metric | BF16 | INT4 |
52
  | -------------- | ------ | ------ |
53
+ | Avg. | 0.4989 | 0.5021 |
54
+ | mmlu | 0.2473 | 0.2456 |
55
  | lambada_openai | 0.6858 | 0.6949 |
56
  | hellaswag | 0.5247 | 0.5177 |
57
+ | winogrande | 0.6480 | 0.6448 |
58
  | piqa | 0.7590 | 0.7573 |
59
  | truthfulqa_mc1 | 0.1971 | 0.2056 |
60
  | openbookqa | 0.2680 | 0.2780 |
61
+ | boolq | 0.6584 | 0.6801 |
62
  | arc_easy | 0.6713 | 0.6717 |
63
  | arc_challenge | 0.3294 | 0.3251 |
64