ModelCloud

/

Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1

like 16

Follow

ModelCloud.AI 70

Is this a marlin kernel compatible?

Yes, but you need to use GPTQModel.load() and pass the parameter backend=BACKEND.MARLIN. It will use the Marlin kernel compatible with group_size 32.