README.md

---
title: Multi-GGUF LLM Inference
emoji: 🧠
colorFrom: pink
colorTo: purple
sdk: streamlit
sdk_version: 1.44.1
app_file: app.py
pinned: false
license: apache-2.0
short_description: Run GGUF models with llama.cpp
---

This Streamlit app enables **chat-based inference** on various GGUF models using `llama.cpp` and `llama-cpp-python`.

### 🔄 Supported Models:
- `Qwen/Qwen2.5-7B-Instruct-GGUF` → `qwen2.5-7b-instruct-q2_k.gguf`
- `unsloth/gemma-3-4b-it-GGUF` → `gemma-3-4b-it-Q4_K_M.gguf`
- `unsloth/Phi-4-mini-instruct-GGUF` → `Phi-4-mini-instruct-Q4_K_M.gguf`
- `MaziyarPanahi/Meta-Llama-3.1-8B-Instruct-GGUF` → `Meta-Llama-3.1-8B-Instruct.Q2_K.gguf`
- `unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF` → `DeepSeek-R1-Distill-Llama-8B-Q2_K.gguf`
- `MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF` → `Mistral-7B-Instruct-v0.3.IQ3_XS.gguf`
- `Qwen/Qwen2.5-Coder-7B-Instruct-GGUF` → `qwen2.5-coder-7b-instruct-q2_k.gguf`

### ⚙️ Features:
- Model selection in the sidebar
- Customizable system prompt and generation parameters
- Chat-style UI with streaming responses
- **Markdown output rendering** for readable, styled output
- **DeepSeek-compatible `<think>` tag handling** — shows model reasoning in a collapsible expander

### 🧠 Memory-Safe Design (for HuggingFace Spaces):
- Loads only **one model at a time** to prevent memory bloat
- Utilizes **manual unloading and `gc.collect()`** to free memory when switching models
- Adjusts `n_ctx` context length to operate within a 16 GB RAM limit
- Automatically downloads models as needed
- Limits history to the **last 8 user-assistant turns** to prevent context overflow

Ideal for deploying multiple GGUF chat models on **free-tier HuggingFace Spaces**!

Refer to the configuration guide at https://huggingface.co/docs/hub/spaces-config-reference