OpenMOSE
/

HRWKV7-Reka-Flash3-Preview

Text Generation

linear-attention

knowledge-distillation

Model card Files Files and versions

OpenMOSE commited on Jul 6

Commit

3bb3693

·

verified ·

1 Parent(s): bfe5add

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -35,8 +35,8 @@ HRWKV7-Reka-Flash3-Preview is an experimental hybrid architecture model that com
 The model implements several key improvements over standard RWKV architectures:
-1. **Token Shift Removal**: Unlike traditional RWKV, the hxa079 variant removes token shifting mechanisms
-2. **GroupNorm Removal**: Eliminates GroupNorm for training stability
 3. **k_first Introduction**: Experimentally adopted the approach of residually connecting k layers in layer 0.
 ### Hybrid Design Benefits

 The model implements several key improvements over standard RWKV architectures:
+1. **Token Shift Removal**: In order to effectively inherit the teacher model weights, we removed the residual connection one token ago.
+2. **GroupNorm Removal**: Helps improve training stability issues
 3. **k_first Introduction**: Experimentally adopted the approach of residually connecting k layers in layer 0.
 ### Hybrid Design Benefits