Upload generation_config.json
#1
by
lucyknada
- opened
No description provided.
quick explainer: without it exl just keeps spamming tokens and the EOS isnt properly set, I'll commit this to all your other repos too
I'm not seeing an issue with token spam? Using my 8.0bpw quant.
Also, the EOS and BOS token IDs in this added config file seem to match what's already in config.json.
if you're not using tabby you might not have that issue; tabby ignores config etc and prefers gen_config; dev was made aware of this issue, but there's no fix for it yet, so it spams all the way to context-size and inbetween puts an .assistant, old llama issue, with gen config it works properly