title: Why nltk word_tokenize is not working even after doing a nltk.download and all the packages are installed correctly?
body: <p>I am using python 3.7 64 bit. nltk version 3.4.5.</p>
  
  <p>When I try to convert text6 in nltk.book to tokens using word_tokenize, I am getting error.</p>
  
  <pre><code>import nltk
  from nltk.tokenize import word_tokenize
  from nltk.book import *
  tokens=word_tokenize(text6)
  </code></pre>
  
  <p>code is done in idle 3.7</p>
  
  <p>Below is the error when I execute the last statement.</p>
  
  <pre><code>    Traceback (most recent call last):
    File "&lt;pyshell#4&gt;", line 1, in &lt;module&gt;
      tokens=word_tokenize(text6)
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\__init__.py", line 144, in word_tokenize
      sentences = [text] if preserve_line else sent_tokenize(text, language)
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\__init__.py", line 106, in sent_tokenize
      return tokenizer.tokenize(text)
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1277, in tokenize
      return list(self.sentences_from_text(text, realign_boundaries))
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1331, in sentences_from_text
      return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1331, in &lt;listcomp&gt;
      return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1321, in span_tokenize
      for sl in slices:
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1362, in _realign_boundaries
      for sl1, sl2 in _pair_iter(slices):
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 318, in _pair_iter
      prev = next(it)
    File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\site-packages\nltk\tokenize\punkt.py", line 1335, in _slices_from_text
      for match in self._lang_vars.period_context_re().finditer(text):
  TypeError: expected string or bytes-like object
  </code></pre>
  
  <p>Please help. Thanks in advance.</p>
  
  <p>While doing some troubleshooting I have created a sample nltk.text.Text object and tried to tokenize it with nltk.word_tokenize. Still I am getting the same error. Please see the below screenshot.<a href="https://i.stack.imgur.com/AXhDV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AXhDV.png" alt="enter image description here"></a></p>
  
  <p>But while calling the nltk.word_tokenize() on string, its working.</p>
  
  <pre><code>&gt;&gt;&gt; tt="Python is a programming language"
  &gt;&gt;&gt; tokens2=nltk.word_tokenize(tt) #Not throwing error
  &gt;&gt;&gt; type(tt)
  &lt;class 'str'&gt;
  &gt;&gt;&gt; type(text6)
  &lt;class 'nltk.text.Text'&gt;
  &gt;&gt;&gt; 
  </code></pre>