eaddario/Llama-Guard-3-8B-GGUF

GGUF and "i-matrix" quantized versions of meta-llama/Llama-Guard-3-8B

Using LLaMA C++ release b4730 for quantization.

Original model: meta-llama/Llama-Guard-3-8B

From the original model creators:

Llama Guard 3 is a Llama-3.1-8B pretrained model, fine-tuned for content safety classification. Similar to previous versions, it can be used to classify content in both LLM inputs (prompt classification) and in LLM responses (response classification). It acts as an LLM – it generates text in its output that indicates whether a given prompt or response is safe or unsafe, and if unsafe, it also lists the content categories violated.

Llama Guard 3 was aligned to safeguard against the MLCommons standardized hazards taxonomy and designed to support Llama 3.1 capabilities. Specifically, it provides content moderation in 8 languages, and was optimized to support safety and security for search and code interpreter tool calls.

All quantized versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration.

At its core, an Importance Matrix (imatrix) is a table or, more broadly, a structured representation that scores the relative importance of different features or parameters in a machine learning model. It essentially quantifies the "impact" each feature has on a specific outcome, prediction, or relationship being modeled.

The process to produce the quantized GGUF models is roughly as follows:

1. Convert the the original model's safetensors into GGUF F16*
2. Estimate the Perplexity score for the F16 model (base) using wikitext-2-raw-v1, and record the logits
3. Generate the imatrix for each calibration dataset
4. Create quantized versions of the base model using each imatrix per quant type
5. Calculate the Perplexity and KL Divergence scores for each quantized model (scores)
6. For each quant type, keep the version with the best (usually the lowest) scores

*BF16 would be preferred, but Apple's GPUs don't support it yet, and therefore any operations are executed in the CPU, making it unacceptably slow. This is expected to change in the near term but until then, if you are using Apple kit avoid using any models tagged BF16

Motivation

An area of ongoing personal research is to optimize the inference performance of LLMs when deployed in resource-constrained environments like, for example, commodity hardware, personal desktops/laptops, edge devices, etc.

The process of quantization reduces the precision of the model's weights, leading to significant reductions in model size, memory needs and computational requirements (a good thing), but this however comes at the expense of a loss in the model's capabilities and accuracy (a bad thing!).

By producing imatrix optimized quantized models, we can maintain inference efficiency whilst reducing memory size and CPU/GPU processing requirements. This optimization is crucial for deploying LLMs on devices with limited hardware capabilities, such as mobile phones or edge devices, without sacrificing significant accuracy.

Models

Filename	Quant type	Size	Perplexity (μ)	ln(PPL(Q)/PPL(base))	KL Divergence (μ)	Description
Llama-Guard-3-8B-F16	F16	16.10G	6.554978 ±0.040159	N/A	N/A	16-bit standard IEEE 754 half-precision floating-point number
Llama-Guard-3-8B-Q8_0	Q8_0	8.54G	6.557455 ±0.040211	99.96%	0.001610 ±0.000012	Extremely high quality, generally unneeded but max available quant
Llama-Guard-3-8B-Q6_K	Q6_K	6.60G	6.570115 ±0.040324	99.90%	0.004383 ±0.000032	Very high quality, near perfect, recommended
Llama-Guard-3-8B-Q5_K_M	Q5_K_M	5.73G	6.611877 ±0.040700	99.79%	0.009900 ±0.000071	High quality
Llama-Guard-3-8B-Q5_K_S	Q5_K_S	5.60G	7.996693 ±0.049993	99.76%	0.011434 ±0.000081	High quality, recommended
Llama-Guard-3-8B-IQ4_NL	IQ4_NL	4.68G	6.763095 ±0.041790	99.26%	0.034552 ±0.000246	Good quality, new method (super-blocks with 256 weights), recommended
Llama-Guard-3-8B-Q4_K_M	Q4_K_M	4.92G	6.727317 ±0.041420	99.40%	0.027914 ±0.000205	Good quality, default size for must use cases, recommended
Llama-Guard-3-8B-Q4_K_S	Q4_K_S	4.69G	6.754714 ±0.041611	99.29%	0.033124 ±0.000235	Good quality, best choice in the Q4 series if RAM is scarce, recommended
Llama-Guard-3-8B-IQ3_M	IQ3_M	3.78G	7.296410 ±0.044494	97.80%	0.103618 ±0.000568	Medium-low quality, new method with decent performance comparable to Q3_K_M
Llama-Guard-3-8B-IQ3_S	IQ3_S	3.68G	7.301537 ±0.044767	97.66%	0.108870 ±0.000594	Lower quality, new method with decent performance, sligthly better than Q3_K_S
Llama-Guard-3-8B-Q3_K_L	Q3_K_L	4.32G	6.983531 ±0.043462	98.37%	0.071002 ±0.000463	Lower quality but usable, good for low RAM availability
Llama-Guard-3-8B-Q3_K_M	Q3_K_M	4.02G	7.058128 ±0.043917	98.13%	0.082098 ±0.000506	Medium-low quality
Llama-Guard-3-8B-Q3_K_S	Q3_K_S	3.66G	7.996693 ±0.049993	95.62%	0.192568 ±0.000912	Lower quality but may be usable in certain cases

I find that quantizations below Q3/IQ3 are not fit for my purposes and therefore do not usually generate them, but happy to provide other quants on request.

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the orignal model the better, thus the closest to 0 the better.

Credits

A big Thank You! to Colin Kealty for the many contributions and for being one of the best sources of high quality quantized models available in Hugginface, and a really big Thank You! to Georgi Gerganov for his amazing work with llama.cpp and the gguf file format.