---
license: apache-2.0
library_name: transformers
base_model:
- Qwen/QwQ-32B
base_model_relation: "adapter"
---



This repo contains the AttnGates' weights for QwQ-32B Model introduced by SeerAttention.

[SeerAttention](https://arxiv.org/pdf/2410.13276) introduces learnable AttnGate modules to accelerate the computationally intensive prefill stage of long-context large language models (LLMs) via dynamic block-level sparsity. The AttnGates are trained in a parameter-efficient self-distillation framework, where they learn to mimic the 2D max-pooled attention patterns of the original frozen model, preserving its integrity while avoiding costly retraining. During inference, these gates generate block-sparse binary masks by applying threshold/TopK to their learned soft scores, enabling efficient computation through a custom block-sparse FlashAttention kernel. 

## Original Github Repo
[https://github.com/microsoft/SeerAttention](https://github.com/microsoft/SeerAttention).