Reproduce pass@10 (temperature 0.8, top p 0.95)
#18
by
sh0416
- opened
Is there anyone who reproduce this metric? I successfully reproduce pass@1 in BigCode leaderboard, but pass@10 in the original paper isn't.
The paper said 0.59, but my experimental result is 0.491.