OpenMOSE
/

HRWKV7-hxa079-Qwen3-14B

Model card Files Files and versions

OpenMOSE commited on Jun 26

Commit

8cbd715

·

verified ·

1 Parent(s): c9ac977

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -18,8 +18,8 @@ HRWKV7-hxa079-Qwen3-14B is an experimental hybrid architecture model that combin
 - **Architecture:** RWKV v7 based "hxa079" Architecture + Group Query Attention Hybrid
 - **Total Layers:** 40 layers (L40D5120)
-  - 34 RWKV layers
-  - 6 GQA layers
 - **Hidden Dimension:** 5120
 - **Training Context Window:** 4096 tokens

 - **Architecture:** RWKV v7 based "hxa079" Architecture + Group Query Attention Hybrid
 - **Total Layers:** 40 layers (L40D5120)
+  - 34 RWKV layers (with Rope)
+  - 6 GQA layers (No Rope, No Position Embeddings)
 - **Hidden Dimension:** 5120
 - **Training Context Window:** 4096 tokens