Blank_Slate_AI / tokenizer_config.json
A-Rink1's picture
Create tokenizer_config.json
b1ae5b6 verified
raw
history blame contribute delete
141 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("base-model-name")
tokenizer.save_pretrained("./my_model")