please create a quantized version, preferably using bitsandbytes!

#3
by ctranslate2-4you - opened

Really like the model but would like to use it with BitsandBytes...

How do we perform inference on a pair of image + prompt, using gguf?

Hey @ctranslate2-4you , check this out: https://huggingface.co/allenai/olmOCR-7B-0225-preview-GGUF

Nice, but I prefer to use BNB for now. do you guys plan on making one? Otherwise, I'd have to add a bunch of llama.cpp dependencies just for this..

Hey @ctranslate2-4you , check this out: https://huggingface.co/allenai/olmOCR-7B-0225-preview-GGUF

Nice, but I prefer to use BNB for now. do you guys plan on making one? Otherwise, I'd have to add a bunch of llama.cpp dependencies just for this..

Noo, not atm. If you plan to make one, can you push it to HF.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment