File size: 3,637 Bytes
c9ac977 8cbd715 c9ac977 57ba57e c9ac977 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
---
license: apache-2.0
---
# HRWKV7-hxa079-Qwen3-14B
### Model Description
HRWKV7-hxa079-Qwen3-14B is an experimental hybrid architecture model that combines RWKV v7's linear attention mechanism with Group Query Attention (GQA) layers. Built upon the Qwen3-14B Instruct foundation, this model replaces most Transformer attention blocks with RWKV blocks while strategically maintaining some GQA layers to enhance performance on specific tasks.
- **Developed by:** OpenMOSE
- **Model type:** Hybrid Linear-Attention Language Model
- **Language(s):** Multilingual (inherited from Qwen3-14B)
- **License:** Apache-2.0
- **Base Model:** Qwen3-14B Instruct
- **Year:** 2025
### Architecture Specifications
- **Architecture:** RWKV v7 based "hxa079" Architecture + Group Query Attention Hybrid
- **Total Layers:** 40 layers (L40D5120)
- 34 RWKV layers (with Rope)
- 6 GQA layers (No Rope, No Position Embeddings)
- **Hidden Dimension:** 5120
- **Training Context Window:** 4096 tokens
- **Inference Context Window** 16384 tokens(100% NIAH)
## Technical Innovation
### RWKV "hxa079" Architecture
The model implements several key improvements over standard RWKV architectures:
1. **Token Shift Removal**: Unlike traditional RWKV, the hxa079 variant removes token shifting mechanisms
2. **GroupNorm Removal**: Eliminates GroupNorm layers for training stability
3. **k_first Introduction**: Implements a novel k_first mechanism optimized for attention conversion
### Hybrid Design Benefits
- **Linear Attention Inference**: RWKV blocks enable O(1) memory complexity during inference
- **Enhanced Needle Tasks**: Strategic placement of GQA layers significantly improves performance on needle-in-haystack retrieval tasks, addressing a known limitation of pure linear attention models
- **Implicit Position Encoding**: Interestingly, the model achieves better performance when RoPE (Rotary Position Embedding) is not applied to GQA layers, suggesting that RWKV blocks provide implicit positional encoding capabilities
## Intended Use
This is an **experimental research model** designed to explore hybrid architectures combining linear and quadratic attention mechanisms. It is intended for:
- Research into efficient attention mechanisms
- Benchmarking hybrid architecture performance
- Exploring linear attention limitations and solutions
- Academic and industrial R&D purposes
## Limitations
- **Experimental Status**: This model is in experimental stages and may exhibit unexpected behaviors
- **Context Window**: Limited to 4096 tokens during training, though RWKV architecture theoretically supports longer sequences
- **Performance Variability**: As a hybrid model, performance may vary significantly across different task types
## Training Details
- **Training Context Window:** 4096 tokens
- **Base Model Initialization:** Weights initialized from Qwen3-14B Instruct
- **Architecture Conversion:** Transformer attention blocks systematically replaced with RWKV blocks, except for 6 strategically placed GQA layers
## Evaluation
Performance evaluation is ongoing. The model shows promising results in:
- Maintaining base model capabilities while achieving linear attention efficiency
- Significantly improved needle-in-haystack task performance compared to pure RWKV architectures
- Competitive performance on standard language modeling benchmarks
## Model Card Contact
OpenMOSE - 2025
---
*Note: This is an experimental model. Performance characteristics and behaviors may differ from both pure RWKV and standard Transformer architectures. Users should thoroughly evaluate the model for their specific use cases.* |