Upload tokenizer.json

#2
by ysdede - opened

Please add default tokenizer.json to make ct2 conversion easier as we copy tokenizer while creating ct2 (faster-whisper) versions, eg:

ct2-transformers-converter --model {asr_model['name']} --output_dir {output_dir} --copy_files preprocessor_config.json tokenizer.json --quantization float16 --force
selimc changed pull request status to merged
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment