---

base_model: Qwen/Qwen2.5-14B-Instruct-1M

---
This is a quantization of the [Qwen2.5-14B-Instruct-1M](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-1M).

Qwen2.5-14B-Instruct-1M, developed by Alibaba Cloud, is a standout model in the world of large language models due to its exceptional capability to handle ultra-long contexts, supporting up to 1 million tokens. This feature makes it significantly more effective for long-context tasks compared to previous versions, while still maintaining strong performance on shorter tasks. With an architecture incorporating advanced techniques like RoPE, SwiGLU, and RMSNorm, Qwen2.5-1M offers a balanced blend of sophistication and efficiency. It is designed as a causal language model and demonstrates considerable prowess in generating coherent and contextually aware text, marking a substantial advancement in handling complex language generation tasks.
## Evaluations
This model provides an accuracy recovery of 100.09%. 

| __English__   |   __[Qwen2.5-14B-Instruct-1M](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-1M)__ |   __[Qwen2.5-14B-Instruct-1M-FP8-Dynamic (this)](https://huggingface.co/cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic)__ |
|:--------------|-------------------------------------------------------------------------------------:|-----------------------------------------------------------------------------------------------------------------------:|
| Avg.          |                                                                                74.79 |                                                                                                                  74.86 |
| ARC           |                                                                                70    |                                                                                                                  70.3  |
| Hellaswag     |                                                                                74.6  |                                                                                                                  74.5  |
| MMLU          |                                                                                79.77 |                                                                                                                  79.77 |

We did not check for data contamination.
     Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`. 
    
## Usage
Install **vLLM** and 
    run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
    
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic --max-model-len 42000 --gpu-memory-utilization 0.95
```
Access the model:
```
curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic",
        "prompt": "San Francisco is a"
    } '
```
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**4497 tokens per second** using one NVIDIA L40S ⚡