Enhance model card: Add metadata, paper/code links, and Transformers usage (#1)

- Enhance model card: Add metadata, paper/code links, and Transformers usage (b5bdfb3f74c076186fff4cc18bb6c0df19930130)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -1,6 +1,19 @@
 ---
 license: apache-2.0
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+  - text-generation
+  - causal-lm
+  - linear-attention
+  - rwkv
+  - reka
+  - knowledge-distillation
+  - multilingual
+languages:
+  - mul
 ---
+
 # HRWKV7-Reka-Flash3-Preview
 
 <div align="center">
@@ -11,27 +24,33 @@ license: apache-2.0
 > It's still far from perfect,
 > but I hope you'll bear with me as I continue this journey. :)
 
+## Paper and Project Details
+
+This model is part of the research presented in the paper [RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale](https://huggingface.co/papers/2505.03005).
+
+The main codebase for the RADLADS project can be found at: [https://github.com/recursal/RADLADS-paper](https://github.com/recursal/RADLADS-paper)
+
 ### Model Description
 
 HRWKV7-Reka-Flash3-Preview is an experimental hybrid architecture model that combines RWKV v7's linear attention mechanism with Group Query Attention (GQA) layers. Built upon the Reka-flash3 21B foundation, this model replaces most Transformer attention blocks with RWKV blocks while strategically maintaining some GQA layers to enhance performance on specific tasks.
 
-- **Developed by:** OpenMOSE
-- **Model type:** Hybrid Linear-Attention Language Model
-- **Language(s):** Multilingual (inherited from Reka-flash3 21B)
-- **License:** Apache-2.0
-- **Base Model:** Reka-flash3 21B(https://huggingface.co/RekaAI/reka-flash-3)
-- **Year:** 2025
+-   **Developed by:** OpenMOSE
+-   **Model type:** Hybrid Linear-Attention Language Model
+-   **Language(s):** Multilingual (inherited from Reka-flash3 21B)
+-   **License:** Apache-2.0
+-   **Base Model:** Reka-flash3 21B(https://huggingface.co/RekaAI/reka-flash-3)
+-   **Year:** 2025
 
 ### Architecture Specifications
 
-- **Architecture:** RWKV v7 based "hxa079" Architecture + Group Query Attention Hybrid
-- **Total Layers:** 44 layers (L44D6114)
-  - 38 RWKV layers (with Rope)
-  - 6 GQA layers (No Rope, No Position Embeddings)
-- **Hidden Dimension:** 6144
-- **Training Context Window:** 4096 tokens
-- **Inference Context Window** 32768+
-- **Training Strategy** Following RADLADS method based knowledge distillation
+-   **Architecture:** RWKV v7 based "hxa079" Architecture + Group Query Attention Hybrid
+-   **Total Layers:** 44 layers (L44D6114)
+    -   38 RWKV layers (with Rope)
+    -   6 GQA layers (No Rope, No Position Embeddings)
+-   **Hidden Dimension:** 6144
+-   **Training Context Window:** 4096 tokens
+-   **Inference Context Window** 32768+
+-   **Training Strategy** Following RADLADS method based knowledge distillation
 
 ## Technical Innovation
 
@@ -39,60 +58,84 @@ HRWKV7-Reka-Flash3-Preview is an experimental hybrid architecture model that com
 
 The model implements several key improvements over standard RWKV architectures:
 
-1. **Token Shift Removal**: In order to effectively inherit the teacher model weights, we removed the residual connection one token ago.
-2. **GroupNorm Removal**: Helps improve training stability issues
-3. **k_first Introduction**: Experimentally adopted the approach of residually connecting k layers in layer 0.
+1.  **Token Shift Removal**: In order to effectively inherit the teacher model weights, we removed the residual connection one token ago.
+2.  **GroupNorm Removal**: Helps improve training stability issues
+3.  **k_first Introduction**: Experimentally adopted the approach of residually connecting k layers in layer 0.
 
 ### Hybrid Design Benefits
 
-- **Linear Attention Inference**: RWKV blocks enable O(1) memory complexity during inference, and the hybrid approach reduces the KVCache to 1/7 of full GQA.
-- **Enhanced Needle Tasks**: Strategic placement of GQA layers significantly improves performance on needle-in-haystack retrieval tasks, addressing a known limitation of pure linear attention models
-- **Implicit Position Encoding**: Interestingly, the model achieves better performance when RoPE (Rotary Position Embedding) is not applied to GQA layers, suggesting that RWKV blocks provide implicit positional encoding capabilities
+-   **Linear Attention Inference**: RWKV blocks enable O(1) memory complexity during inference, and the hybrid approach reduces the KVCache to 1/7 of full GQA.
+-   **Enhanced Needle Tasks**: Strategic placement of GQA layers significantly improves performance on needle-in-haystack retrieval tasks, addressing a known limitation of pure linear attention models
+-   **Implicit Position Encoding**: Interestingly, the model achieves better performance when RoPE (Rotary Position Embedding) is not applied to GQA layers, suggesting that RWKV blocks provide implicit positional encoding capabilities
 
 ## Intended Use
 
 This is an **experimental research model** designed to explore hybrid architectures combining linear and quadratic attention mechanisms. It is intended for:
 
-- Research into efficient attention mechanisms
-- Benchmarking hybrid architecture performance
-- Exploring linear attention limitations and solutions
-- Academic and industrial R&D purposes
+-   Research into efficient attention mechanisms
+-   Benchmarking hybrid architecture performance
+-   Exploring linear attention limitations and solutions
+-   Academic and industrial R&D purposes
 
 ## Limitations
 
-- **Experimental Status**: This model is in experimental stages and may exhibit unexpected behaviors
-- **Context Window**: Limited to 4096 tokens during training, though RWKV architecture theoretically supports longer sequences
-- **Performance Variability**: As a hybrid model, performance may vary significantly across different task types
+-   **Experimental Status**: This model is in experimental stages and may exhibit unexpected behaviors
+-   **Context Window**: Limited to 4096 tokens during training, though RWKV architecture theoretically supports longer sequences
+-   **Performance Variability**: As a hybrid model, performance may vary significantly across different task types
 
 ## Training Details
 
-- **Training Context Window:** 4096 tokens
-- **Training GPU** AMD MI300X x 1(takes 68hrs)
-- **Training Strategy** 8bit MLP Quant, frozen emb,mlp,head, Deepspeed Stage1
-- **Base Model Initialization:** Weights initialized from Reka-flash3 21B
-- **Architecture Conversion:** Transformer attention blocks systematically replaced with RWKV blocks, except for 6 strategically placed GQA layers
+-   **Training Context Window:** 4096 tokens
+-   **Training GPU** AMD MI300X x 1(takes 68hrs)
+-   **Training Strategy** 8bit MLP Quant, frozen emb,mlp,head, Deepspeed Stage1
+-   **Base Model Initialization:** Weights initialized from Reka-flash3 21B
+-   **Architecture Conversion:** Transformer attention blocks systematically replaced with RWKV blocks, except for 6 strategically placed GQA layers
 
 ## Evaluation
 
 Performance evaluation is ongoing. The model shows promising results in:
-- Maintaining base model capabilities while achieving linear attention efficiency
-- Significantly improved needle-in-haystack task performance compared to pure RWKV architectures
-- Competitive performance on standard language modeling benchmarks
-
+-   Maintaining base model capabilities while achieving linear attention efficiency
+-   Significantly improved needle-in-haystack task performance compared to pure RWKV architectures
+-   Competitive performance on standard language modeling benchmarks
+
+## Usage with Hugging Face Transformers
+
+This model can be loaded and used with the `transformers` library. Ensure you have `transformers` installed: `pip install transformers`.
+When loading, remember to set `trust_remote_code=True` because of the custom architecture.
+
+```python
+from transformers import pipeline, AutoTokenizer
+import torch
+
+model_name = "OpenMOSE/HRWKV7-Reka-Flash3-Preview" # Replace with the actual model ID if different
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+pipe = pipeline(
+    "text-generation",
+    model_name,
+    tokenizer=tokenizer,
+    torch_dtype=torch.bfloat16, # or torch.float16 depending on your GPU and model precision
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+text = "The quick brown fox jumps over the lazy "
+result = pipe(text, max_new_tokens=20, do_sample=True, top_p=0.9, temperature=0.7)[0]["generated_text"]
+print(result)
+```
 
-## Run
-- RWKV-Infer now support hxa079
+## Run with RWKV-Infer (as provided by original authors)
+-   RWKV-Infer now support hxa079
 ```bash
-curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"/home/client/Projects/llm/hxa079-reka-flash3-stage2-hybrid.pth","model_viewname":"RWKV HXA079 L38T6 Reka Flash3","model_strategy":"int8","adapter_filename":"","adapter_mode":"", "template":"rekaflash3", "endtoken":"\n <sep>","default_temperature":"0.2", "default_top_p":"0.3", "rope_theta":"8000000.0", "rms_norm_eps":"1e-5"}'
+curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"/home/client/Projects/llm/hxa079-reka-flash3-stage2-hybrid.pth","model_viewname":"RWKV HXA079 L38T6 Reka Flash3","model_strategy":"int8","adapter_filename":"","adapter_mode":"", "template":"rekaflash3", "endtoken":"
+ <sep>","default_temperature":"0.2", "default_top_p":"0.3", "rope_theta":"8000000.0", "rms_norm_eps":"1e-5"}'
 
 ```
 
 ## Thank you for Big help :)
-- SmerkyG Inspired by RADLADS (https://arxiv.org/abs/2505.03005)
-- https://github.com/recursal/RADLADS-paper
+-   SmerkyG Inspired by RADLADS (https://arxiv.org/abs/2505.03005)
 
 ## Training Code
-- https://github.com/OpenMOSE/RWKVInside (still buggy)
+-   https://github.com/OpenMOSE/RWKVInside (still buggy)
 
 
 ## Model Card Contact
@@ -101,4 +144,20 @@ OpenMOSE - 2025
 
 ---
 
-*Note: This is an experimental model. Performance characteristics and behaviors may differ from both pure RWKV and standard Transformer architectures. Users should thoroughly evaluate the model for their specific use cases.*
+*Note: This is an experimental model. Performance characteristics and behaviors may differ from both pure RWKV and standard Transformer architectures. Users should thoroughly evaluate the model for their specific use cases.*
+
+## Citation
+
+If you use this code or find our work valuable, please consider citing RADLADS:
+
+```bibtex
+@misc{goldstein2025radladsrapidattentiondistillation,
+      title={RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale}, 
+      author={Daniel Goldstein and Eric Alcaide and Janna Lu and Eugene Cheah},
+      year={2025},
+      eprint={2505.03005},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2505.03005}, 
+}
+```