|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- HuggingFaceFW/fineweb-edu |
|
|
--- |
|
|
|
|
|
|
|
|
# SparseModernBERT α=1.5 Model Card |
|
|
|
|
|
## Model Overview |
|
|
|
|
|
SparseModernBERT-alpha1.5 is a masked language model based on [ModernBERT](https://github.com/AnswerDotAI/ModernBERT) that replaces the standard softmax attention with an adaptive sparse attention mechanism (AdaSplash) using Triton. |
|
|
|
|
|
The sparsity parameter α = 1.5 yields moderately sparse attention patterns, improving efficiency while maintaining performance. |
|
|
|
|
|
**Key features:** |
|
|
|
|
|
* **Sparsity (α)**: 1.5 |
|
|
* **Tokenization**: same as ModernBERT |
|
|
* **Pretraining**: masked language modeling on a large web corpus |
|
|
|
|
|
## Usage |
|
|
|
|
|
|
|
|
Use the codebase from: https://github.com/deep-spin/SparseModernBERT |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer |
|
|
from sparse_modern_bert import CustomModernBertModel |
|
|
|
|
|
model_id = "sardinelab/SparseModernBERT-alpha1.5" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = CustomModernBertModel.from_pretrained(model_id, trust_remote_code=True) |
|
|
``` |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in your work, please cite: |
|
|
|
|
|
```bibtex |
|
|
@article{goncalves2025adasplash, |
|
|
title={AdaSplash: Adaptive Sparse Flash Attention}, |
|
|
author={Gon\c{c}alves, Nuno and Treviso, Marcos and Martins, Andr\'e F. T.}, |
|
|
journal={arXiv preprint arXiv:2502.12082}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
|
|
|
|
|
|
