Spaces:

---

Duplicated from  datalawyer/legal_pt_llm_leaderboard

eduagarcia

/

open_pt_llm_leaderboard

like 227

Running on CPU Upgrade

Hi.

We were checking the results on the leaderboard for Falcon-H1 models, which unfortunately are much lower than what we expected based on our internal benchmarks. Amongst other things, we noticed that the results don't correlate well with model size, and that many scores are 0.

I would like to discuss some potential issues that might have affected the scores, and hopefully we can debug this together. I would also like to ask if it would be possible to rerun the evaluation after understanding the cause of the low scores, as we believe the present scores are not representative of the models' true capabilities.

1. When checking the generation files specifically for Falcon3-H1-34B, for example in:
   https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/blob/main/tiiuae/Falcon-H1-34B-Base/raw_2025-07-15T04-30-48.728728/pretrained__tiiuae__Falcon-H1-34B-Base%2Cdtype__bfloat16%2Cdevice__cuda%3A0%2Crevision__main%2Ctrust_remote_code__True%2Cmax_gen_toks__1536%2Cstarting_max_length__2560_enem_challenge.jsonl
   I saw the following parameters:
   "original_fewshots_size": 3,
   "effective_fewshots_size": 0,
   Hence it is unclear if the evaluation for ENEM was run using 3-shots or 0-shot. Also, it was my understanding that all models should be run under a few-shot scenario on the leaderboard. Could you please confirm if the evaluations for this model were run as 0-shot or few-shot?

2. In the above case I also saw that the generated output was:
   "resps": [
   [
   "\n"
   ]
   ]
   It seems the generation was stopped before the model could actually output any useful tokens. I tried to reproduce this under different scenarios:

• under 0-shot, my tests show that the generation indeed starts with a "\n" followed by the useful token (a letter A/B/C/D).
• under 3-shot, my tests showed the model already produced " letter", as expected.
• in no case did the model produce just "\n", hence either the generation was interrupted or there was a postprocessing issue cuting the answer before the useful token.

3. In some instances, we noticed that the generation was:
   "resps": [
   [
   " N"
   ]
   ]
   instead of "Não", or
   "resps": [
   [
   " Pos/Neg/Ne"
   ]
   ]
   instead of "Positivo"/"Negativo"/"Neutro". We again suspect that maybe the generation is incomplete or truncated.
   Here are some examples where the above issues can be found:
   https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/blob/main/tiiuae/Falcon-H1-7B-Base/raw_2025-07-09T15-41-50.552612/pretrained__tiiuae__Falcon-H1-7B-Base%2Cdtype__bfloat16%2Cdevice__cuda%3A0%2Crevision__main%2Ctrust_remote_code__True%2Cmax_gen_toks__1536%2Cstarting_max_length__2560_assin2_rte.jsonl
   https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/blob/main/tiiuae/Falcon-H1-3B-Base/raw_2025-07-09T04-02-33.419627/pretrained__tiiuae__Falcon-H1-3B-Base%2Cdtype__bfloat16%2Cdevice__cuda%3A0%2Crevision__main%2Ctrust_remote_code__True%2Cmax_gen_toks__1536%2Cstarting_max_length__2560_tweetsentbr.jsonl
   https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/blob/main/tiiuae/Falcon-H1-0.5B-Base/raw_2025-07-08T05-07-27.156126/pretrained__tiiuae__Falcon-H1-0.5B-Base%2Cdtype__bfloat16%2Cdevice__cuda%3A0%2Crevision__main%2Ctrust_remote_code__True%2Cmax_gen_toks__1536%2Cstarting_max_length__2560_tweetsentbr.jsonl

4. To try to replicate the mentiond issues, I pulled the fork of lm-eval: https://github.com/eduagarcia/lm-evaluation-harness-pt
   Then I chose a smaller model tiiuae/Falcon-H1-0.5B-Base that had scored 0 on tweetSentBR. I ran this command:
   lm_eval --model hf --model_args pretrained=tiiuae/Falcon-H1-0.5B-Base,trust_remote_code=True,apply_chat_template=False --tasks tweetsentbr --device cuda:0 --batch_size auto --output_path results/tiiuae/Falcon-H1-0.5B-Base --log_samples
   The results and samples are available here: https://huggingface.co/datasets/rcojocaru/eval-results-Falcon-H1-0.5B-Base-tweetsentbr/tree/main
   The score I got was higher (54) and the generations looked complete, with no truncation. This reinforces my belief that some issue plagued most of the evaluations of the H1 models, maybe related to the lm-eval branch or the version of transformers used? In my case I used the main branch of the form and transformers version '4.55.3'.

Please let me know your feedback on the above issues when possible. Thank you!

Best regards,
Ruxandra