Commit
·
dbc7a14
1
Parent(s):
1ff8888
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
|
|
|
|
|
| 1 |
This is https://huggingface.co/NeelNanda/gpt-neox-tokenizer-digits but with a fix that makes this work on `tokenizers == 0.14` after a breaking change involving `added_tokens`: https://github.com/huggingface/tokenizers/issues/1358
|
| 2 |
|
| 3 |
The two changes from NeelNanda/gpt-neox-tokenizer-digits are
|
|
|
|
| 1 |
+
TODO: Probably we need to add the spaces to the end of the vocab!!! Despite being useless!!!
|
| 2 |
+
|
| 3 |
This is https://huggingface.co/NeelNanda/gpt-neox-tokenizer-digits but with a fix that makes this work on `tokenizers == 0.14` after a breaking change involving `added_tokens`: https://github.com/huggingface/tokenizers/issues/1358
|
| 4 |
|
| 5 |
The two changes from NeelNanda/gpt-neox-tokenizer-digits are
|