DeepSeek V3 AWQ

AWQ of the DeepSeek V3 chat model.

This quant modified some of the model code to fix the overflow issue when using float16.

Tested on vLLM with 8x H100, inference speed 5 tokens per second with batch size 1 and short prompt, 12 tokens per second when using moe_wna16 kernel.

Downloads last month: 4,229

Inference Providers NEW

Text Generation

This model is not currently available via any of the supported third-party Inference Providers, and HF Inference API does not yet support model repos that contain custom code.

Model tree for cognitivecomputations/DeepSeek-V3-AWQ

Base model

deepseek-ai/DeepSeek-V3

Quantized

(12)

this model