| This is a double fine-tuned version of Mistral Small 24B Base 2501. | |
| Stage 1 was shoving 30M tokens of human-writen story content into it using completion training ([ToastyPigeon/ms3-base-roselily](https://huggingface.co/ToastyPigeon/ms3-base-roselily)), which is about half of my WIP Roselily dataset (~60M tokens total). | |
| Stage 2 was teaching it instruct (this model). | |
| This model should accept (in theory) any of the following instruct formats: | |
| **Tekken v7** | |
| ``` | |
| [SYSTEM_PROMPT]{system prompt}[/SYSTEM_PROMPT][INST]{user message}[/INST]{assistant response}</s> | |
| ``` | |
| **ChatML** | |
| ``` | |
| <|im_start|>system | |
| {system prompt}<|im_end|> | |
| <|im_start|>user | |
| {user message}<|im_end|> | |
| <|im_start|>assistant | |
| {assistant response}<|im_end|> | |
| ``` | |
| **Fizzpaca** | |
| ``` | |
| ### System: | |
| {system prompt} | |
| ### Instruction: | |
| {user message} | |
| ### Response: | |
| {assistant response}</s> | |
| ``` | |
| The Tekken tokens were already in the tokenizer. unused special tokens #20 and 21 were repurposed for the ChatML tokens. Fizzpaca did not add any. | |
| You may need to add both `</s>` and `<|im_end|>` as stop tokens for it to work properly with all formats. |