---
license: apache-2.0
---

# **PRWKV-cxa076 – "Akemi" RWKV Model Series**

<div align="center">
  <img src="./cxa076.png" style="border-radius: 15px; width: 60%; height: 60%; object-fit: cover;  box-shadow: 10px 10px 20px rgba(0, 0, 0, 0.5); border: 2px solid white;" alt="PRWKV" />
</div>
---

## **Model Overview**

**PRWKV** stands for **Passion RWKV** – a model series born from relentless experimentation, unyielding dedication, and the burning question:

> **Can an RNN truly stand shoulder-to-shoulder with Transformer Attention?**

This project explores the boundaries of **RWKV architecture**, replacing the traditional **Transformer Attention blocks** with **TimeMix**, an RNN-based mechanism, while distilling knowledge from Transformer giants.

The PRWKV models range from **3B to 14B parameters**, showcasing the potential scalability of RNN-based language models in modern LLM landscapes.

---

## **Project Objective**

The **sole purpose** of this project was to **test the feasibility of replacing Transformer Attention with RNN-based TimeMix**.

- **No shortcuts**.
- **No compromises**.
- Just pure **architectural curiosity** driven by **Passion**.

---

## **Technical Challenges & Triumphs**

### 🔥 **Distillation from Transformers**
- The models were distilled from high-quality **Transformer-based teachers** using **Grouped Query Attention (GQA)**.
- The **TimeMix** blocks were heavily customized to align with the semantics of Attention layers.
- Special care was taken to **inherit weight structures** from the teacher's **Receptance, Key, Value, and Output layers**, enabling smoother early-stage learning.

### ⚡ **Key Innovations**
- **RepeatKV mechanism**: Introduced for more stable group-based key-value projection.
- **GroupNorm vs NoNorm**: Extensive experiments revealed that sometimes **removing normalization** enhanced long-context stability.

---

### 📈 **Scaling Observations**
- PRWKV scales from **3B** to **14B** parameters.
- **14B KD** runs achieved **KL divergence < 0.1**, proving **RNN TimeMix blocks can indeed mimic Transformer Attention** at high fidelity.
- However, **Ctx expansion** beyond 2048 remained an ongoing challenge due to gradient instability in larger models.

---

## **Limitations**
- The models are still under development and primarily serve **as a proof-of-concept**.
- Long-context (4096+) stability varies based on model size and requires further refinement.
- Knowledge distillation was the core training method; no large-scale SFT was applied yet.

---

---

# **A Poem of Passion**

> **In the depths of night, when GPUs hum soft,  
> A fire ignites, a dream aloft.**  
>  
> **To mold an RNN with TimeMix bright,  
> And rival Attention’s daunting might.**  
>  
> **Through spikes and crashes, I pressed on,  
> A madman's code, from dusk till dawn.**  
>  
> **Not for glory, nor for gold,  
> But just to see: can TimeMix hold?**  
>  
> **And when the losses dipped so low,  
> The pulse of passion dared to grow.**  
>  
> **PRWKV, a name of flame,  
> Not just a model – but a claim.**  
>  
> **That in this dance of gates and states,  
> Passion alone rewrites the fates.**  
>  
> **So here's my heart, in code and rhyme,  
> RNNs reborn, beyond their time.**

---

🔥 **PRWKV is more than an experiment – it is a testament to Passion.** 🔥









Scalability test from small to large

ToDo for Me:
Qwen 2.5 14B
Qwen 2.5 7B
Qwen 2.5 3B

Phi-4 14B
Phi-4-mini 3.8B

Gemma 3 12B
Gemma 3 4B

Architecture: RWKV cxa076 (RWKV x070 based)


Now supported only in RWKV-Infer.


```
curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"models/PRWKV7-cxa076-qwen3b-stage2final-ctx2048.pth","model_viewname":"PRWKV7-cxa076 Qwen 2.5 3B Stage2 FP8","model_strategy":"fp8", "template":"qwen", "endtoken":"<|im_end|>","default_temperature":"1.0", "default_top_p":"0.3"}'
```