File size: 2,294 Bytes
30058fe |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
---
license: gemma
library_name: transformers
pipeline_tag: text-generation
tags:
- conversational
- gguf
- llamacpp
---
# Gemma 2 9b Instruction Tuned - GGUF
These are GGUF quants of [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it)
Details about the model can be found at the above model page.
## Llamacpp Version
These quants were made with llamacpp tag b3408.
If you have problems loading these models, please update your software to se the latest llamacpp version.
## Perplexity Scoring
Below are the perplexity scores for the GGUF models. A lower score is better.
| Quant Level | Perplexity Score | Standard Deviation |
|-------------|------------------|--------------------|
| F32 | 8.7849 | 0.06498 |
| BF16 | 8.7849 | 0.06498 |
| Q8_0 | 8.7869 | 0.06500 |
| Q6_K | 8.7972 | 0.06510 |
| Q5_K_M | 8.7791 | 0.06489 |
| Q5_K_S | 8.7899 | 0.06503 |
| Q4_K_M | 8.8745 | 0.06575 |
| Q4_K_S | 8.9293 | 0.06636 |
| Q3_K_L | 9.0210 | 0.06693 |
| Q3_K_M | 9.1213 | 0.06784 |
| Q3_K_S | 9.1857 | 0.06726 |
## Quant Details
This is the script used for quantization.
```bash
#!/bin/bash
# Define MODEL_NAME above the loop
MODEL_NAME="gemma-2-9b-it"
# Define the output directory
outputDir="${MODEL_NAME}-GGUF"
# Create the output directory if it doesn't exist
mkdir -p "${outputDir}"
# Make the F32 quant
f32file="${outputDir}/${MODEL_NAME}-F32.gguf"
if [ -f "${f32file}" ]; then
echo "Skipping f32 as ${f32file} already exists."
else
python convert_hf_to_gguf.py "~/src/models/${MODEL_NAME}" --outfile "${f32file}" --outtype "f32"
fi
# Abort out if the F32 didn't work
if [ ! -f "${f32file}" ]; then
echo "No ${f32file} found."
exit 1
fi
# Define the array of quantization strings
quants=("Q8_0" "Q6_K" "Q5_K_M" "Q5_K_S" "Q4_K_M" "Q4_K_S" "Q3_K_L" "Q3_K_M" "Q3_K_S")
# Loop through the quants array
for quant in "${quants[@]}"; do
outfile="${outputDir}/${MODEL_NAME}-${quant}.gguf"
# Check if the outfile already exists
if [ -f "${outfile}" ]; then
echo "Skipping ${quant} as ${outfile} already exists."
else
# Run the command with the current quant string
./llama-quantize "${f32file}" "${outfile}" "${quant}"
echo "Processed ${quant} and generated ${outfile}"
fi
done
```
|