Hugging Face's logo

bigcode
/
santacoder

Text Generation
Transformers
PyTorch
code
gpt2
custom_code
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
45

Switch from PreTrainedTokenizerFast to GPT2TokenizerFast and add eos_token & bos_token

#15
by loubnabnl HF staff - opened
base: refs/heads/main
←
from: refs/pr/15
Discussion Files changed
+4
-2
loubnabnl
 BigCode org

PreTrainedTokenizerFast returns token_type_ids by default and santacoder is not trained on them so passing model(tokenizer(text)) can result in weird behavior in some cases. We'll use GPT2TokenizerFastinstead.

Switch from PreTrainedTokenizerFast to GPT2TokenizerFast and add eos_token & bos_tokend63779a6
loubnabnl changed pull request status to merged
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment