Sokrates9
/

texttest

Text Classification

Model card Files Files and versions Community

texttest / tokenizer.json

Sokrates9's picture

Create tokenizer.json

d3e6afd verified about 1 month ago

history blame contribute delete

157 Bytes

	from transformers inport AutoTokenizer

	model_name = "meta-llama/Llama-3-8B"
	tokenizer = AutoTokenizer.from_pretrained(model_name)
	tokenizer.save_pretrained