how to make determinstic output?
I run the model multiple times with the same input but the outputs are different.
This is expected by default. Does anyone know how to produce deterministic output?
In addition to (temperature=0, top_p =1)
Try setting a seed value (eg: seed=0).
For purely deterministic, I always use Temp=0, Top_K=0, Repeat_Penalty=1, Min_P=1, Top_P=0.
For Seed number you can set it to whatever value you like, but I recommend using the maximum "Highly Composite Number" that your environment supports.
Link to OEIS article for Highly Composite Numbers. https://oeis.org/A002182
For KoboldCPP this number is "720720", and for LM Studio this number is "4488062423933088000". LM Studio does allow you to use exponents, so you can try "9.14876906094457e+256" but it might not give the best answer compared to the other number.
@Koitenshin
,
I'm using vllm, but still can't get deterministic output with following configuration.
Could you please help to provide suggestion, especially for the seed?
"seed": 42, # should be a very large number?
"temperature": 0,
"n": 1,
"top_p": 0.01, # top_p must be in (0, 1]
"extra_body": {
"top_k": 1,
"min_p": 1,
"repeat_penalty": 1,
},
@tonyaw
top_p & top_k should have been "0", I have no idea what "n" means because I either use LMStudio (which really sucks), or KoboldCPP (but I'll probably be switching to croco.cpp soon)
And for seed, you can use whatever you want. I just really recommend "Highly Composite Numbers" because there's magic in the math.