tulip-tokenizer / tokenizer.json
ZinengTang's picture
Update tokenizer by filtering out foreign language tokens
3abc7d3 verified
This file is stored with Git LFS . It is too big to display, but you can still download it.

Git LFS Details

  • SHA256: 79c526bf76110757b7a04320ff23126cb14690a4c485b837ecc4956ffcc10c1d
  • Pointer size: 133 Bytes
  • Size of remote file: 14.3 MB

Git Large File Storage (LFS) replaces large files with text pointers inside Git, while storing the file contents on a remote server. More info.