---
license: apache-2.0
---

# **Model Card: PRWKV-7-Qwen3-14B-Preview-v0.1**

### **Overview**

- **Model Name:** PRWKV-7-Qwen3-14B-Preview-v0.1  
- **Base Model:** Qwen3 14B (Instruct)
- **Architecture:** RWKV Cxa076r (RWKV x070 Based) + SwiGLU  
- **Parameter Count:** 14 Billion  
- **Context Length:** 3072  
- **Training Tokens:**  
  - Stage 1: 100 Million Tokens  
  - Stage 2: 200 Million Tokens  

This model is part of an experimental effort to *replace Transformer-style attention with a fully recurrent RWKV-based architecture*. It uses a customized version of the RWKV TimeMix block (`Cxa076r`) with SwiGLU activation, applied to a 14B-scale model derived from Qwen3.

---

### **Motivation**

The goal of this project is to explore whether an RNN-style model such as RWKV can faithfully mimic the output and reasoning behavior of large Transformer-based LLMs like Qwen3, while retaining the benefits of linear compute cost and persistent memory.

Replacing attention with TimeMix was not a trivial task. Qwen3 is heavily optimized for attention-based flow, including grouped-query attention (GQA) and Rotary Positional Embeddings (RoPE). To bridge the architecture gap, we introduced novel gating structures, careful initialization alignment, and staged distillation involving both token-level and hidden-state mimicry.

---

### **Challenges Faced**

- **Stability in Early Training:**  
  Unlike Transformer models, RWKV's state dynamics require careful gating and normalization. Without it, token dropout or state explosion frequently occurred during warm-up.
  
- **Cross-Architecture Distillation:**  
  Aligning a recurrent architecture with a feed-forward Transformer introduced step-wise divergence, especially in conversational jumps. Custom loss functions were employed to match hidden trajectories and long-term behavior, not just per-token outputs.
  
- **Context Sensitivity:**  
  Increasing context length beyond 2048 revealed stability cliffs. Careful adjustment of temporal decay, positional mixing, and memory routing was necessary to reach 3072 tokens reliably.

---

### **Current Limitations**

This is a *preview* version. The model is capable of coherent generation, especially in long-form settings, but may still show deviations in precision-demanding tasks or rare contexts. Prompt injection robustness and RLHF alignment are future work.

---

### **License & Usage**

This model is intended for **research and experimentation only**. Please consult the licensing terms of Qwen3 and RWKV if you intend to use this model commercially or fine-tune it.

---

### **Poem – The Cost of Curiosity**

> Countless times we failed—  
> A ghost in the gradients,  
> A silence in the state.  
>  
> Attention was easy.  
> But ease never leads to breakthrough.  
>  
> We drank too much coffee.  
> Slept too little.  
>  
> And somewhere between the hallucinations,  
> The loss spikes,  
> And the whispered curses at 3am—  
>  
> A new mind was born.  
> PRWKV-7 lives.

---

2025 OpenMOSE

https://x.com/_m0se_