Update README.md

README.md

@@ -11,6 +11,33 @@ license: cc-by-sa-4.0
 
 The Micka Tokenizer is a subword tokenizer with a vocabulary size of 32,768 tokens, designed to support a wide array of natural language processing (NLP) tasks in both Slovenian and English.
 
+It includes a series of special tokens tailored for handling various structured language tasks, from standard natural language processing needs to more specific chatbot and conversational models. Special unicode characters are used as delimiters (`⸢` and `⸥`) at the start and end of each special token, reducing the likelihood that these token sequences will appear in regular text data.
+- **Start delimiter**: `⸢` (Unicode: `U+2E22`)
+- **End delimiter**: `⸥` (Unicode: `U+2E25`)
+
+## Special Tokens
+
+The tokenizer supports a range of special tokens that enable structured processing, padding, and segmentation in different tasks:
+
+- **Padding, Masking, and Separation Tokens**:
+  - **Padding** (`⸢PAD⸥`): Used to pad sequences to a uniform length.
+  - **Masking** (`⸢MSK⸥`): Marks parts of a sequence that should be masked for certain tasks, such as masked language modeling.
+  - **Separation** (`⸢|⸥`): A general-purpose separator token.
+
+- **Sentence and Paragraph Structure Tokens**:
+  - **Start of Sentence** (`⸢s⸥`) and **End of Sentence** (`⸢/s⸥`): Define sentence boundaries for better structuring and parsing within long text sequences.
+  - **Start of Paragraph** (`⸢p⸥`) and **End of Paragraph** (`⸢/p⸥`): Define paragraph breaks, which can be helpful for document-based tasks or summarization tasks where paragraph structure is essential.
+
+- **Out-of-Vocabulary / Unknown Token**:
+  - **Unknown Token** (`⸢UNK⸥`): Represents out-of-vocabulary words or unknown tokens that the model encounters.
+
+- **Chatbot and Conversational Tokens**:
+  - **System Message Start** (`⸢SYS⸥`): Marks the beginning of a system message, useful in multi-turn dialogue systems.
+  - **User Message Start** (`⸢USR⸥`): Designates the start of a user’s input, enabling clear differentiation in conversation logs.
+  - **Agent Message Start** (`⸢AGT⸥`): Indicates the start of a response from the chatbot or conversational agent.
+
+
+
 ## Model Details
 
 - **Developed by**: Marko Kokol