--- license: apache-2.0 language: - en pipeline_tag: conversational tags: - not-for-all-audiences base_model: mistralai/Mistral-7B-Instruct-v0.2 --- # ShoriRP 🏆 LIMA-like model mostly based on data from two RP forums as well as some synthetically crafted chats from [Limamono](https://huggingface.co/lemonilia/Limamono-Mistral-7B-v0.50). An important difference from LimaRP, other than the subject focus, is that conversations are multi-character where applicable, wheras LimaRP only included 1-on-1 RP. Furthermore, the messages sampled have shorter length in general. The rationale behind this was that the short(er)-form roleplays are more "fun" on average, while the longer ones tend to use common purple prose tropes and be a bit dull. **This is still a work in progress. Updates will be posted in the future.** # Technical details - Part of the data was semi-automatically grammar-checked using Mixtral-Instruct (3.5 bpw). - The prose has been homogenized to a consistent novel-like format with narration in third person and past tense. - OOC was intentionally _not_ completely eliminated, and its format was homogenized to a single one `(OOC: this is a message.)`. Likewise, URLs have not been all deleted unless they referred to internal forum resources. - For a very small portion of the data, dialogue lines and thoughts, suitable emoji (mostly 1, up to 3) conveying the mood have been _prepended_. _Prepending_ instead of _appending_ helps the model and the reader to prepare for the message tone. - Usernames have been entirely removed; only character names remained in the data (same policy as with LimaRP). # Known issues - Some character names might be overrepresented in the dataset. - Occasionally at the beginning of the chat there might be impersonation issues. This could be due to still insufficient amount of data or not training the model enough. - There might be some residual "alignment" from the base model used for finetuning. # Prose format All training samples use book (novel) format with narration in third person / past tense. Other formats are not supported (they might work, but not consistently). ## Details - Character thoughts are delimited with underscores `_`. - Onomatopoeias are delimited with single asterisks `*`. - Emphasized text is delimited by double asterisks `**`. - Spoken dialogues are delimited with ASCII quote marks `"`. - Non-dialogue quotes are replaced with double apostrophes `''`. This avoids distracting and/or annoying conflicts with the dialogue highlighting in SillyTavern. - Text to be interpreted as louder than normal is in `ALL CAPS`. - Quoted text from other people is most of the time prepended with `>`. - Formatted output text is delimited with triple backticks ` ``` `, sometimes followed by additional identifiers specifying the language (markdown, text, etc). # Prompting format Suitable `json` files have been provided to easily apply the prompting format in SillyTavern. - [Context](https://huggingface.co/lemonilia/test-model-srp-0.5/blob/main/BlockML-Context.json) - [Instruct](https://huggingface.co/lemonilia/test-model-srp-0.5/resolve/main/BlockML-Instruct.json) Note: the prompting format is **intentionally different** from that of the Mistral-Instruct base model. It is advised to use `▀` as a stop token. ## Block characters The model uses a _ChatML-like_ prompting format with a few changes from the usual roles typically used for ChatGPT-like assistant chatbots. The first is that to avoid complications with tokenization and special tokens, `<|im_start|>` has been replaced with `▀` (upper half block character) and `<|im_end|>` has been replaced with `▄` (lower half block character). Both of these tokens already exist in the Mistral tokenizer as single tokens; they don't have any combination with other tokens, nor any special meaning attached to them, so for all intents and purposes they work like special tokens. This avoids complications related with training a model with new tokens, as well tokenization issues that occur with ChatML tokens when used literally. ## Roles In summary, these are the tags in the typical order they would appear: - `title` - `tags` - `lore` - `description` - `scenario` - `message` - `OOC` (not added in the training data yet) ### message Messages are all under the `message` role regardless of who writes it. The rationale is that since conversations are multi-character and the characters do not necessarily reply in a fixed order, it won't be possible to reliably establish who is the "human" in terms of training. `message` was found to be neutral enough as a role and a better fit, considering the length hints that shuld be added. Like LimaRP, messages use **length hints**. It's recommended to always add them, otherwise the model may output very short messages. The available lengths are: `nano`, `micro`, `tiny`, `short`, `medium`, `long`, `massive`, `huge`, `enormous` The recommended length is _medium_. The longest sizes do not have a large amount of training data, so they might not work very reliably. ### title **Optional**. The title of the roleplay. It's a role for steering the conversation at the beginning. If added, this should be in the first message block. ### tags **Optional**. A list of comma-separated releavant tags to hint the model about the chat contents. If added, it should be after the title. ### lore **Optional**. Background lore or past story is to be placed under the `lore` role, without length hints. ### scenario **Optional**. Future events that must still happen go in `scenario`. ### description This is where character cards go. No specific layout for character profiles is defined, but the name of the character should be clear from the description. In the training data, profiles may occasionally appear mid-conversation (for example when a new character appears). Try to use one `description` message block per character. ## Example prompt template ```text ▀title {story title} ▀tags {comma-separated list of tags} ▀lore {{loreBefore}}▄ ▀description {{char}} {{description}} ▀description {{user}} {{persona}} ▀scenario {what will happen in the beginning of the roleplay}▄ ▀message (length: {length}) {{char}}: {message}▄ ▀message (length: {length}) {{user}}: {message}▄ ▀message (length: {length}) {{char}}: {message}▄ [...] ``` ## Practical example ``` ▀title A strange incident in Gensokyo▄ ▀tags barrier, danmaku, magic, reimu, marisa▄ ▀description **Name:** Reimu Hakurei **Age:** 18 **Personality:** Calm and collected. She is a very responsible person and tries to do her job as well as she can. She also likes to take care of people around her, even if they are not always nice to her. **Appearance:** Reimu is a young girl with long, black hair and brown eyes. She wears a red ribbon and matching tubes on her sidelocks and a traditional shrine maiden uniform, with a red hakama over a white kimono. **Background:** Reimu is the shrine maiden of Hakurei Shrine, located in the center of Gensokyo. She spends most of her time taking care of the shrine and performing various duties for the residents of the village. She is known to be quite skilled in the use of magic, especially when it comes to barrier magic.▄ ▀description **Name:** Marisa Kirisame **Personality:** Impulsive and energetic. She is often seen as a troublemaker by others due to her tendency to break rules and cause chaos wherever she goes. She is also a bit of a flirt and enjoys teasing others.▄ ▀message (length: medium) Reimu: "Hmm... I wonder what's going on?" Reimu mused as she stood at the entrance to the shrine, looking out at the village beyond. It was unusually quiet today, with no one coming to visit or offer any kind of offering. She had been expecting a few visitors this morning, but none had shown up yet. "Maybe everyone is busy with something else today? Or maybe they're all sick?" she thought as she turned back inside, closing the door behind her. She began tidying up the shrine, making sure everything was clean and ready for visitors. As she worked, she couldn't shake the feeling that something wasn't right.▄ ▀message (length: short) Marisa: "Ooohh! Reimu-chan~!" Marisa suddenly appeared from nowhere, landing on the ground with a soft thud. "What's wrong? Why aren't there any customers today? Aren't you supposed to have lots of visitors every day? I thought you were famous for being able to heal injuries and cure diseases..." She gave her friend a wink before continuing, "But I guess I could always come by and give you some company! I'm bored anyway~"▄ ▀message (length: long) Reimu: _Ugh, that girl again..._ Reimu thought as she looked at Marisa with annoyance. The younger girl was known for causing mischief wherever she went, and Reimu didn't appreciate her interrupting her work. "I don't know, Marisa," she replied curtly. "No one seems to be coming today. Maybe they're all busy with their own things. But thank you for offering your help." Reimu continued cleaning the shrine while keeping an eye on Marisa. She knew that if she left the girl alone for too long, she would probably start causing trouble. She just hoped that nothing bad happened today.▄ ``` # Dataset Similar to LimaRP, but more niche. Flexible training sample length (from 4k to 32k tokens, at least). Might or might not be released in the future. ## Message length distribution Most messages are below 300 tokens in length. ![Message length distribution](https://files.catbox.moe/1w7pdr.png) # Training details ## Hardware 1x NVidia RTX 3090 24GB ## Software [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) ## Training hyperparameters ```yaml load_in_4bit: true adapter: qlora sequence_len: 16384 sample_packing: true pad_to_sequence_len: false gradient_accumulation_steps: 4 micro_batch_size: 1 eval_batch_size: 1 num_epochs: 2 optimizer: adamw_bnb_8bit lr_scheduler: constant learning_rate: 0.000085 weight_decay: 0.05 train_on_inputs: true bf16: true fp16: false tf32: true lora_r: 20 lora_alpha: 16 lora_dropout: 0.1 lora_target_linear: true ```