Italian ModernBERT

Model Description

Italian ModernBERT (DeepMount00/Italian-ModernBERT-base) is a specialized Italian language version of ModernBERT, specifically pre-trained on Italian text corpora. This model is designed exclusively for Italian language tasks.

Key Features

Architecture: Based on ModernBERT-base (22 layers, 149M parameters)
Context Length: 8,192 tokens
Language: Italian-only
Tokenizer: Custom tokenizer optimized for Italian language
Training: Pre-trained on Italian text corpus

Technical Details

Uses Rotary Positional Embeddings (RoPE)
Implements Local-Global Alternating Attention
Supports Flash Attention 2 for optimal performance
No token type IDs required

Limitations

Optimized only for Italian language processing
Not suitable for other languages
May reflect biases present in training data