FP8 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

DeepSeek-R1-Distill Quantized

Granite 3.1 Quantization

Sparse-Llama-3.1-2of4

Vision Language Models Quantization

FP8 LLMs for vLLM

Llama-3.2 Quantization

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

FP8 LLMs for vLLM

updated Oct 17, 2024

Accurate FP8 quantized models by Neural Magic, ready for use with vLLM!

neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 990 • 31
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 43.3k • 40
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8

Text Generation • Updated 1 day ago • 354k • 41
neuralmagic/Mistral-Nemo-Instruct-2407-FP8

Text Generation • Updated Jul 19, 2024 • 26.3k • 18
neuralmagic/Meta-Llama-3-8B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 10.6k • 23
neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 217 • 14
neuralmagic/Meta-Llama-3-70B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 25.6k • 11
neuralmagic/Mixtral-8x7B-Instruct-v0.1-AutoFP8

Text Generation • Updated Jul 18, 2024 • 1.25k • 3
neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV

Text Generation • Updated Jun 19, 2024 • 12.7k • 7
neuralmagic/Meta-Llama-3-70B-Instruct-FP8-KV

Text Generation • Updated Jun 26, 2024 • 188 • 2
neuralmagic/Mixtral-8x22B-Instruct-v0.1-AutoFP8

Text Generation • Updated Aug 12, 2024 • 136 • 3
neuralmagic/Qwen2-72B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 471 • 14
neuralmagic/Qwen2-7B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 6.51k • 1
neuralmagic/Qwen2-1.5B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 4.68k
neuralmagic/Qwen2-0.5B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 6.81k • 3
neuralmagic/Mistral-7B-Instruct-v0.3-FP8

Text Generation • Updated Jul 18, 2024 • 556 • 2
neuralmagic/Llama-2-7b-chat-hf-FP8

Text Generation • Updated Jul 18, 2024 • 306
neuralmagic/gemma-2-9b-it-FP8

Text Generation • Updated Jul 18, 2024 • 806 • 6
neuralmagic/Qwen2-57B-A14B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 482 • 1
neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 7.94k • 6
neuralmagic/DeepSeek-Coder-V2-Lite-Base-FP8

Text Generation • Updated Jul 18, 2024 • 10
neuralmagic/DeepSeek-Coder-V2-Base-FP8

Text Generation • Updated Jul 22, 2024 • 7
neuralmagic/DeepSeek-Coder-V2-Instruct-FP8

Text Generation • Updated Jul 22, 2024 • 1.06k • 7
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 10.6k • 5
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 4.68k • 6
neuralmagic/Meta-Llama-3.1-8B-FP8

Text Generation • Updated 1 day ago • 5.91k • 6
neuralmagic/Meta-Llama-3.1-70B-FP8

Text Generation • Updated Oct 9, 2024 • 591 • 1
neuralmagic/starcoder2-15b-FP8

Text Generation • Updated Oct 9, 2024 • 228
neuralmagic/starcoder2-3b-FP8

Text Generation • Updated Oct 9, 2024 • 80
neuralmagic/starcoder2-7b-FP8

Text Generation • Updated Oct 9, 2024 • 5
neuralmagic/Meta-Llama-3.1-405B-FP8

Text Generation • Updated Oct 9, 2024 • 126
neuralmagic/gemma-2-2b-it-FP8

Updated Oct 9, 2024 • 1.66k • 1
neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic

Text Generation • Updated Oct 9, 2024 • 50.2k • 3
neuralmagic/Llama-3.2-3B-Instruct-FP8-dynamic

Text Generation • Updated Oct 9, 2024 • 34.5k • 2
neuralmagic/Llama-3.2-3B-Instruct-FP8

Text Generation • Updated Oct 16, 2024 • 41k • 6
neuralmagic/Llama-3.2-1B-Instruct-FP8

Text Generation • Updated Oct 16, 2024 • 9.29k • 2
neuralmagic/Llama-3.2-1B-FP8

Updated Oct 9, 2024 • 70
neuralmagic/Phi-3.5-mini-instruct-FP8-KV

Text Generation • Updated Oct 1, 2024 • 817 • 2
neuralmagic/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic

Text Generation • Updated Oct 17, 2024 • 1.62k • 14