C4ai-command-r-plus Tokenizing?
Been alternating between running this model with DeepSeek's tokenizer and Qwen2 tokenizer, both seem to produce strong results - but different ones.
However, I noticed in longer contexts focused on story or RP, Lamarck starts spitting out tokens I've only seen from CohereForAI/c4ai-command-r-plus?
Namely <|CHATBOT_TOKEN|>, and sometimes <act>, which I think is just logical vomit of think->act. But the Cohere token is strange, because I don't see any of that model in any part of the merge history of Lamarck?
Does another model in Lamarck's history use this token?
Nevermind, there was a problem with my UI and some wires got crossed, false alarm. >.>
Despite your fix being somewhere else. I'm glad to hear Lamarck is holding up well with different tokenizers. This has been on my radar. Thank you for the feedback!
To clarify, one of my setting got changed from Deekseek-R1 to Command-R, which caused the <|CHATBOT_TOKEN|> to get used - I didn't notice because Lamarck was still giving good output! Except for the in output after which was being hidden by my formatting.
Not sure why <|CHATBOT_TOKEN|> leads to <act> but otherwise that format was working as well.