File size: 2,042 Bytes
122dd56 5fcc5dc 122dd56 6e08c30 942f441 122dd56 a62af3e 122dd56 a2274c6 6e08c30 a2274c6 41d5365 a2274c6 6e08c30 a2274c6 122dd56 8a769b2 122dd56 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
---
license: apache-2.0
---
# RWKV-x070-2B9-CJE-Instruct Model Card
## Model Overview
- **Model Name**: RWKV-x070-2B9-CJE-Instruct
- **Description**: An instruction-tuned model specialized for Japanese, Chinese, and English languages
- **Base Model**: rwkv-x070-2b9-world-v3-40%trained-20250113-ctx4k.pth
- **Architecture**: RWKV x070 "Goose"
- **Parameters**: 2.9B
- **Model Dimension**: 2560
- **Number of Layers**: 32
## Fine-tuning Details
### Training Configuration
- **Trainer**: RWKV-LM-RLHF (https://github.com/OpenMOSE/RWKV-LM-RLHF)
- **PEFT Mode**: Hybrid Training combining frozen embeddings and Bone (Block Affine Transformation) + full parameter training
- **SFT Method**: SmoothingLoss SFT
- **Context Window**: 5120 (trained with 1024 token overlap)
- **Compute Power**: AMD Instinct MI100 x 2 60hrs (100％ solar energy)
### Dataset Specifications
- **Size**: 800k pairs
- **Content**:
- Mixed data in Japanese, Chinese, and English
- Conversations
- Programming code
- Translation tasks
- Chain-of-Thought reasoning tasks
### How to use
- Install latest RWKV-Infer (Linux,WSL) (https://github.com/OpenMOSE/RWKV-Infer)
- make folder 'models'
- move rwkv-x070-2b9-cje-instruct-1.pth to models folder
```
curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"models/rwkv-x070-2b9-cje-instruct-1.pth","model_viewname":"RWKV x070 2B9 CJE Instruct-1","model_strategy":"fp16","endtoken":"\\n\\n\\x17"}'
```
- Enjoy with openai compatible api http://127.0.0.1:9000/v1 :)
### Important Note
- Set the end token as '\n\n\x17'
```
User: who are you?\n\n\x17
Assistant: gooday i'm rwkv\n\n\x17
```
### Limitations and Considerations
- This is an experimental model; inference stability is not fully guaranteed
- Unexpected behaviors may occur
- Continuous improvements are being made; feedback is welcome
## License
Apache License 2.0
## Acknowledgments
We express our gratitude to the RWKV base model and the RWKV community for their support in developing this model. |