README.md

metadata

base_model:
  - huihui-ai/QwQ-32B-abliterated

vllm (pretrained=/root/autodl-tmp/QwQ-32B,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.432	±	0.0314
		strict-match	5	exact_match	↑	0.744	±	0.0277

vllm (pretrained=/root/autodl-tmp/QwQ-32B,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.444	±	0.0222
		strict-match	5	exact_match	↑	0.716	±	0.0202

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.8140	±	0.0125
- humanities	2	none		acc	↑	0.8359	±	0.0251
- other	2	none		acc	↑	0.8103	±	0.0269
- social sciences	2	none		acc	↑	0.8889	±	0.0222
- stem	2	none		acc	↑	0.7544	±	0.0238

vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.528	±	0.0316
		strict-match	5	exact_match	↑	0.740	±	0.0278

vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.492	±	0.0224
		strict-match	5	exact_match	↑	0.742	±	0.0196

vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated,add_bos_token=true,max_model_len=4096,dtype=bfloat16,max_num_seqs=3), gen_kwargs: (None), limit: 15.0, num_fewshot: None, batch_size: 1

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.8152	±	0.0126
- humanities	2	none		acc	↑	0.8359	±	0.0253
- other	2	none		acc	↑	0.8000	±	0.0276
- social sciences	2	none		acc	↑	0.8722	±	0.0240
- stem	2	none		acc	↑	0.7754	±	0.0232

vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated-awq,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.476	±	0.0316
		strict-match	5	exact_match	↑	0.752	±	0.0274

vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated-awq,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.524	±	0.0224
		strict-match	5	exact_match	↑	0.716	±	0.0202

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.8023	±	0.0130
- humanities	2	none		acc	↑	0.8000	±	0.0266
- other	2	none		acc	↑	0.7949	±	0.0284
- social sciences	2	none		acc	↑	0.8500	±	0.0258
- stem	2	none		acc	↑	0.7789	±	0.0235