ZinengTang
/

tulip-tokenizer

Model card Files Files and versions Community

tulip-tokenizer

1 contributor

History: 2 commits

ZinengTang's picture

Update tokenizer by filtering out foreign language tokens

b3b3dbb verified 7 days ago

.gitattributes

1.57 kB

Update tokenizer by filtering out foreign language tokens 7 days ago
tokenizer.json

43.6 MB
LFS

Update tokenizer by filtering out foreign language tokens 7 days ago
tokenizer.model

4.24 MB
LFS

Update tokenizer by filtering out foreign language tokens 7 days ago
tokenizer_config.json

46.5 kB

Update tokenizer by filtering out foreign language tokens 7 days ago