xprophetnet-large-wiki100-cased / tokenizer_config.json
lysandre's picture
lysandre HF staff
Adds tokenizer_config.json file
78b61f0 verified
raw
history blame
183 Bytes
{"bos_token": "[SEP]", "eos_token": "[SEP]", "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "mask_token": "[MASK]", "model_max_length": 512, "do_lower_case": false}