---
license: mit
base_model:
- unsloth/phi-4
library_name: transformers
---
# WAIDWML - What Am I Doing With My Life?
### (8 Phi-4s in a trenchcoat)

## Rationale
So there I was, finding some inspiration to tune stuff but lacking the disposable funds to do anything with the larger models. Enter Phi-4, a model designed for productivity...
Initially it  was just a going to be a sequential series of finetunes, starting from the baseline Phi-4 and gradually adding more datasets until I either got bored or it got good, but then I had an idea; what if I just MoE'd it?

Yeah.

As a proof of concept, this wasn't too bad. The end result is... _interesting_, to say the least.

## Training
As mentioned above, this was done in "phases", each with a separate dataset. Most were done with a `max_seq_length` of 32k, a few of them were dropped to 16k to make sure they fit in the hardware.

`lr` was all over the place but in general somewhere between `1e-5` and `4e-6`. These were all separate LoRAs using `r=64` and `alpha=32` with rsLoRA enabled. `epochs` were 2 or 3 for everything except `c2`, as that'd take far too long.

- `p1`: Private RP dataset (`RPT-Varied-Small`)
- `p2`: `TheDrummer/AmoralQA-v2`
- `p3`: `AIRRC/Eudaimonic`
- `p4`: Two private RP datasets (`cc-gpt4-sfw-sharegpt` & `cc-gpt4-nsfw-sharegpt`)
- `p5`: A random subset of the infamous "`c2`"-logs dataset, cleaned and deduped (approx. 30%)
- `p6`: Private RP dataset (`RPT-Varied-Small_v1.5`)
- `p7`: `NewEden/PIPPA-Mega-Filtered`
- `p8`: `Squish42/bluemoon-fandom-1-1-rp-cleaned`

(Note: the `RPT-Varied-Small` and `RPT-Varied-Small_v1.5` datasets are due to be released after I manually verify their fitness.)

Once all LoRAs were trained, I separately merged them into the base model then I used [mergekit](https://github.com/arcee-ai/mergekit) [(config)](https://huggingface.co/rAIfle/WAIDWML-Phi4-8x14B-bf16/blob/main/mergekit_moe_config.yml) to "merge" them into a MoE. I chose to initialize the router randomly as I was going to training that part later. After that, I trained the routing layers for 8 epochs with `lr = 1e-6` and `grimulkan/LimaRP-augmented` as the dataset. It took roughly 8.5 hours on a 6xA40 instance on RunPod.

## Recommended Settings
Phi-4 format. What I used for my tests:
- Temp 1
- minP 0.05


## FAQ
```
Q: Why not do anything constructive, like GRPO-tune a model of usable size?
A: Where's the fun in that?

Q: Are you, like, okay?
A: Objectively? Probably not. Subjectively? Never better.

Q: You know this still sucks for RP, right?
A: Yup. Should have pivoted to reasoning and code once R1 hit, but sunk cost and all kept me on this trajectory.
```