Noise with 8bit model
On the regular model everything is fine for me.
But with 8bit bnb I get noise. At first I modified the code for bnb myself, installed it and started getting noise. Then someone suggested your repo, I took your files and... I still get noise. =(
With 4bit all is ok!
I yanked the 8 bit for now, I think I either uploaded the wrong folder or the wrong inference code on that one, apologies! When I get a few spare hours I'll re-test and upload again. 4 bit works fine.
Thanks for doing this.. I see there is now a vibevoice for pinokio made I think pretty recently. It seems to have a way to load a "custom model" Im going to try to see if I can load the 4bit even though the 1.5 as is works for me. I would like to see if the 4bit works with this pinokio implementation. Because if it does I think the 8bit would work as well and give me better quality. I have a windows machine with a 4070 12gig card and 128GB system ram. Another nice thing with pinokio is that it runs even better on a linux host machine. Which I will be happy to test as well. I have a linux host with a old 3060 and 48GB system ram.
Well I tried and failed to incorporate the downloads from huggingface into the pinokio directory. It looks like it can "see" the directory but doesnt load the model. Sorry my knowledge on how these models get loaded. It says something about bitsandbytes in the log... Maybe a developer might take a look.. Again thanks for making the quant. I can make generations with the 7b model.. (maybe the ram I have lets me do that. Just takes a really long time. And the 1.5 works but quite often it generates background music. And while I might be able to remove it.. Still remains less ideal to have to do that.