eliebak
/

the-tokenizer-v1.5

Inference Endpoints

Model card Files Files and versions Community

eliebak HF staff commited on Jun 27, 2024

Commit

e418e1e

·

verified ·

1 Parent(s): 4ab427c

Update README.md

Files changed (1) hide show

README.md +3 -1

README.md CHANGED Viewed

@@ -5,7 +5,9 @@ tags: []
 # Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->

 # Model Card for Model ID
+`lvwerra/the-tokenizer-v1` but replacing:
+- <fim_prefix> -> <|im_start|> (token id 1)
+- <fim_middle> -> <|im_end|> (token id 2)