ZinengTang
/

tulip-tokenizer

SHA256: 6b9a4a8e2650f595c38da0ca3640e9fb840d91244bae3bf5a13f53dd75863681
Pointer size: 133 Bytes
Size of remote file: 27.2 MB

tulip-tokenizer / tokenizer.json

Update tokenizer by filtering out foreign language tokens

78436e6 verified about 11 hours ago

27.2 MB

This file is stored with Git LFS . It is too big to display, but you can still download it.

Git Large File Storage (LFS) replaces large files with text pointers inside Git, while storing the file contents on a remote server. More info.