andrewcanis

/

c4ai-command-r-v01-GGUF

like 84

I have the same problem.

You probably need to pull this branch for it to work https://github.com/ggerganov/llama.cpp/pull/6033

I got it working by pulling https://github.com/acanis/llama.cpp/ running make and then

./main --model ./models/c4ai-command-r-v01-Q2_K.gguf --temp 0.8 --reverse-prompt "<|END_OF_TURN_TOKEN|>" --prompt "<BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>"

edit: you might need to add -c 1024 (or higher) as the default context window seems to be 512 tokens

I added a note about this to the README. The PR should be merged upstream soon.

Added a note to the readme: the GGUF files are compatible with llama.cpp from Mar 16, 2024 onwards, starting from release b2440