Upload tokenizer files (vocab, config, README)

README.md

@@ -0,0 +1,20 @@
+---
+language: en
+tags:
+- tokenizer
+- pytorch
+- streaming
+library_name: nano
+---
+
+# Nano Tokenizer
+
+This tokenizer was trained using a Python-only pipeline (no `transformers` or `tokenizers`), on a dataset streamed from the Hugging Face Hub.
+
+## Usage
+
+```python
+from transformers import PreTrainedTokenizerFast
+tokenizer = PreTrainedTokenizerFast.from_pretrained("your-username/tokenizer-name")
+```
+

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "unk_token": "<unk>",
+  "pad_token": "<pad>",
+  "bos_token": "<bos>",
+  "eos_token": "<eos>"
+}

tokenizer_config.json

@@ -1,8 +1,8 @@
 {
+  "model_max_length": 512,
+  "tokenizer_class": "PreTrainedTokenizerFast",
   "unk_token": "<unk>",
   "pad_token": "<pad>",
   "bos_token": "<bos>",
-  "eos_token": "<eos>",
-  "model_max_length": 512,
-  "tokenizer_class": "PreTrainedTokenizerFast"
+  "eos_token": "<eos>"
 }