title: AttributeError: 'NoneType' object has no attribute 'tokenize'
body: <p>I am trying to use XLNET through transformers. however i keep getting the issue &quot;AttributeError: 'NoneType' object has no attribute 'tokenize'&quot;. I am unsure of how to proceed. if anyone could point me in the right direction it would be appreciated.</p>
  <pre><code>tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)
  
  print(' Original: ', X_train[1])
  
  # Print the tweet split into tokens.
  print('Tokenized: ', tokenizer.tokenize(X_train[1]))
  
  # Print the tweet mapped to token ids.
  print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(X_train[1])))
  
  
  
  
  Original:  hey angel duh sexy really thanks haha
  ---------------------------------------------------------------------------
  AttributeError                            Traceback (most recent call last)
  &lt;ipython-input-67-2b1b432b3e15&gt; in &lt;module&gt;()
        2 
        3 # Print the tweet split into tokens.
  ----&gt; 4 print('Tokenized: ', tokenizer.tokenize(X_train[2]))
        5 
        6 # Print the tweet mapped to token ids.
  
  AttributeError: 'NoneType' object has no attribute 'tokenize'
  </code></pre>