title: OSError: Can't load tokenizer
body: <p>I want to train an XLNET language model from scratch. First, I have trained a tokenizer as follows:</p>
  <pre><code>from tokenizers import ByteLevelBPETokenizer
  
  # Initialize a tokenizer
  tokenizer = ByteLevelBPETokenizer()
  # Customize training
  tokenizer.train(files='data.txt', min_frequency=2, special_tokens=[ #defualt vocab size
      &quot;&lt;s&gt;&quot;,
      &quot;&lt;pad&gt;&quot;,
      &quot;&lt;/s&gt;&quot;,
      &quot;&lt;unk&gt;&quot;,
      &quot;&lt;mask&gt;&quot;,
  ])
  tokenizer.save_model(&quot;tokenizer model&quot;)
  </code></pre>
  <p>Finally, I will have two files in the given directory:</p>
  <pre><code>merges.txt
  vocab.json
  </code></pre>
  <p>I have defined the following config for the model:</p>
  <pre><code>from transformers import XLNetConfig, XLNetModel
  config = XLNetConfig()
  </code></pre>
  <p>Now, I want to recreate my tokenizer in transformers:</p>
  <pre><code>from transformers import XLNetTokenizerFast
  
  tokenizer = XLNetTokenizerFast.from_pretrained(&quot;tokenizer model&quot;)
  </code></pre>
  <p>However, the following error appears:</p>
  <pre><code>File &quot;dfgd.py&quot;, line 8, in &lt;module&gt;
      tokenizer = XLNetTokenizerFast.from_pretrained(&quot;tokenizer model&quot;)
    File &quot;C:\Users\DSP\AppData\Roaming\Python\Python37\site-packages\transformers\tokenization_utils_base.py&quot;, line 1777, in from_pretrained
      raise EnvironmentError(msg)
  OSError: Can't load tokenizer for 'tokenizer model'. Make sure that:
  
  - 'tokenizer model' is a correct model identifier listed on 'https://huggingface.co/models'
  
  - or 'tokenizer model' is the correct path to a directory containing relevant tokenizer files
  </code></pre>
  <p>What should I do?</p>