Please add more details to readme
Would you consider to add more info ?It can be like your tweets on x, then i can draw more attention to this work.
Hello!
Of course, I'll continue to improve the Readme!
Currently, I'm working alone on
- Optimizing the Inference engine(focusing on triton flash attention)
- Building an hxa079 fine-tuning framework using RWKV-LM-RLHF(SFT, offline RL etc)
- implementing tests in llama.cpp( its very difficult :( head=96, 128 still not working well )
- Coding HF compatible inference code
and am behind in some areas. please stay tuned :)
readme updated :)
Since this is the first MOE-RWKV hybrid model, could we create a technical report for this project? Also, can we run some benchmarks to evaluate its performance? epicially we have new methods to finetune or RL.
Sorry to bring this up, due to we can make this work more meaningful and impactful
sorry, this is 'not' MoE model. just dense.
but, Qwen3-30BA3B also trained, but due to limitations in the MoE implementation of RWKV-Infer, the inference speed is very slow. I plan to release it after resolving this issue. If you don't mind the slow inference speed, i can release it.