title: NLP: Tokenize : TypeError: expected string or bytes-like object
body: <blockquote>
  <p><strong>I also tried .apply(str) and .astype(str) before tokenization, yet I get TypeError: expected string or bytes-like object.</strong></p>
  </blockquote>
  <hr />
  <pre><code>data.info()
  
  &lt;class 'pandas.core.frame.DataFrame'&gt;
  RangeIndex: 8 entries, 0 to 7
  Data columns (total 3 columns):
   #   Column           Non-Null Count  Dtype 
  ---  ------           --------------  ----- 
   0   tag              8 non-null      object
   1   clean_patterns   8 non-null      object
   2   clean_responses  8 non-null      object
  dtypes: object(3)
  memory usage: 320.0+ bytes
  </code></pre>
  <hr />
  <blockquote>
  <p><strong>I am trying to word_tokenize the data for the NLP chatbot.</strong></p>
  </blockquote>
  <hr />
  <pre><code>print(word_tokenize(data))
  </code></pre>
  <blockquote>
  <p>TypeError                                 Traceback (most recent call
  last)  in 
  ----&gt; 1 print(word_tokenize(data))</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize_<em>init</em>_.py in
  word_tokenize(text, language, preserve_line)
  128     :type preserve_line: bool
  129     &quot;&quot;&quot;
  --&gt; 130     sentences = [text] if preserve_line else sent_tokenize(text, language)
  131     return [
  132         token for sent in sentences for token in _treebank_word_tokenizer.tokenize(sent)</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize_<em>init</em>_.py in
  sent_tokenize(text, language)
  106     &quot;&quot;&quot;
  107     tokenizer = load(&quot;tokenizers/punkt/{0}.pickle&quot;.format(language))
  --&gt; 108     return tokenizer.tokenize(text)
  109
  110</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in tokenize(self,
  text, realign_boundaries)    1272         Given a text, returns a list
  of the sentences in that text.    1273         &quot;&quot;&quot;
  -&gt; 1274         return list(self.sentences_from_text(text, realign_boundaries))    1275     1276     def debug_decisions(self,
  text):</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in
  sentences_from_text(self, text, realign_boundaries)    1326<br />
  follows the period.    1327         &quot;&quot;&quot;
  -&gt; 1328         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]    1329     1330     def _slices_from_text(self,
  text):</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in (.0)
  1326         follows the period.    1327         &quot;&quot;&quot;
  -&gt; 1328         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]    1329     1330     def _slices_from_text(self,
  text):</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in
  span_tokenize(self, text, realign_boundaries)    1316         if
  realign_boundaries:    1317             slices =
  self._realign_boundaries(text, slices)
  -&gt; 1318         for sl in slices:    1319             yield (sl.start, sl.stop)    1320</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in
  _realign_boundaries(self, text, slices)    1357         &quot;&quot;&quot;    1358         realign = 0
  -&gt; 1359         for sl1, sl2 in _pair_iter(slices):    1360             sl1 = slice(sl1.start + realign, sl1.stop)    1361             if not
  sl2:</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in _pair_iter(it)
  314     it = iter(it)
  315     try:
  --&gt; 316         prev = next(it)
  317     except StopIteration:
  318         return</p>
  <p>D:\anaconda\lib\site-packages\nltk\tokenize\punkt.py in
  _slices_from_text(self, text)    1330     def _slices_from_text(self, text):    1331         last_break = 0
  -&gt; 1332         for match in self._lang_vars.period_context_re().finditer(text):    1333<br />
  context = match.group() + match.group(&quot;after_tok&quot;)    1334<br />
  if self.text_contains_sentbreak(context):</p>
  <p>TypeError: expected string or bytes-like object</p>
  </blockquote>