GGUF Broken?
Hi,
the GGUF you've uploaded seems to be broken (?)
But it seems to be an underlying issue with the Yi 1.5 Architecture.
Do you have any ideas?
(I should've been more clear)
The issue lies within the output quality being completely detached from the context and borderline nonsense
If the model loads, it's likely not an issue with GGUF or the conversion, but what you see is the actual model output. Some models are simply unhinged, some are sensitive to sampler settings or prompt format. I am not aware of an issue affecting Yi 1.5 specifically, but it is of course always a possbility.
It likely used the wrong tokenizer after all but there is nothing we can do about it. @Luni You can check https://huggingface.co/mradermacher/model_requests/discussions/305 where we already discussed about this potential issue before quantizing this model. During my testing back then I was satisfied with the output. Maybe because I used it in completion mode. If you use completion mode using latest llama it is probably fine.