---
license: cc-by-nc-4.0
language:
- pl
base_model:
- CYFRAGOVPL/PLLuM-8x7B-nc-instruct
---
# PLLuM-8x7B-nc-instruct GGUF Quantizations by Nondzu

DISCLAIMER: This is state of the art quantized model. I am not the author of the original model. I am only hosting the quantized models. I do not take any responsibility for the models.

This repository contains GGUF quantized versions of the [PLLuM-8x7B-nc-instruct](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-nc-instruct) model. All quantizations were performed using the  [llama.cpp](https://github.com/ggerganov/llama.cpp) (release [b4768](https://github.com/ggml-org/llama.cpp/releases/tag/b4768)). These quantized models can be run in [LM Studio](https://lmstudio.ai/) or any other llama.cpp–based project.

## Prompt Format

Use the following prompt structure:
```
???
```

## Available Files

Below is a list of available quantized model files along with their quantization type, file size, and a short description.

| Filename                                                                              | Quant Type | File Size | Description                                                                                   |
| ------------------------------------------------------------------------------------- | ---------- | --------- | --------------------------------------------------------------------------------------------- |
| [PLLuM-8x7B-nc-instruct-Q2_K.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)    | Q2_K       | 17 GB     | Very low quality but surprisingly usable.                                                   |
| [PLLuM-8x7B-nc-instruct-Q3_K.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)    | Q3_K       | 21 GB     | Low quality, suitable for setups with very limited RAM.                                       |
| [PLLuM-8x7B-nc-instruct-Q3_K_L.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q3_K_L     | 23 GB     | High quality; recommended for quality-focused usage.                                          |
| [PLLuM-8x7B-nc-instruct-Q3_K_M.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q3_K_M     | 21 GB     | Very high quality, near perfect output – recommended.                                         |
| [PLLuM-8x7B-nc-instruct-Q3_K_S.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q3_K_S     | 20 GB     | Moderate quality with improved space efficiency.                                              |
| [PLLuM-8x7B-nc-instruct-Q4_K_M.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q4_K_M     | 27 GB     | Default quality for most use cases – recommended.                                             |
| [PLLuM-8x7B-nc-instruct-Q4_K_S.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q4_K_S     | 25 GB     | Slightly lower quality with enhanced space savings – recommended when size is a priority.       |
| [PLLuM-8x7B-nc-instruct-Q5_K_M.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q5_K_M     | 31 GB     | High quality – recommended.                                                                   |
| [PLLuM-8x7B-nc-instruct-Q5_K_S.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)  | Q5_K_S     | 31 GB     | High quality, offered as an alternative with minimal quality loss.                            |
| [PLLuM-8x7B-nc-instruct-Q6_K.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)    | Q6_K       | 36 GB     | Very high quality with quantized embed/output weights.                                        |
| [PLLuM-8x7B-nc-instruct-Q8_0.gguf](https://huggingface.co/Nondzu/PLLuM-8x7B-chat-GGUF/tree/main)    | Q8_0       | 47 GB     | Maximum quality quantization.                                                                 |

## Downloading Using Hugging Face CLI

<details>
  <summary>Click to view download instructions</summary>

First, ensure you have the Hugging Face CLI installed:

```bash
pip install -U "huggingface_hub[cli]"
```

Then, target a specific file to download:

```bash
huggingface-cli download Nondzu/PLLuM-8x7B-instruct-nc-GGUF --include "PLLuM-8x7B-instruct-nc-Q4_K_M.gguf" --local-dir ./
```

For larger files, you can specify a new local directory (e.g., `PLLuM-8x7B-instruct-nc-Q8_0`) or download them directly into the current directory (`./`).

</details>