title: How to Tokenize a list of lists of lists of strings
body: <p>I have a text dataset that is a list of lists of lists of strings. I need to <code>Tokenize</code> this data to fit it into a classification model. I am very familiar with using <code>keras.preprocessing.text.Tokenizer</code> to do so, and often do this with the below code:</p>
  
  <pre class="lang-py prettyprint-override"><code>data = 
      [[['not'],
        ['ahead'],
        ['um let me think'],
        ['thats not very encouraging if they had a cast of thousands on the other end']],
        [['okay civil liberties tell me your position'],
       ['probably would go ahead']],
       [['oh'],
       ['it up so i dont know where you really go'],
       ['well most of my problem with this latest task'],
       ['its some i kind of dont want to put in the time to do it'],
       ['right so im saying ive got a lot of other things to do']]]
  
  tokenizer = Tokenizer()
  tokenizer.fit_on_texts(data)
  sequences = tokenizer.texts_to_sequences(data)
  </code></pre>
  
  <p>When I run this code on my data, I get the following error:</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>2 frames
  &lt;ipython-input-44-1da804f42cc8&gt; in main()
       12     # tokenize and vectorize text data to prepare for embedding
       13     tokenizer = Tokenizer()
  ---&gt; 14     tokenizer.fit_on_texts(new_corpus)
       15     sequences = tokenizer.texts_to_sequences(new_corpus)
       16     word_index = tokenizer.word_index
  
  /usr/local/lib/python3.6/dist-packages/keras_preprocessing/text.py in fit_on_texts(self, texts)
      213                 if self.lower:
      214                     if isinstance(text, list):
  --&gt; 215                         text = [text_elem.lower() for text_elem in text]
      216                     else:
      217                         text = text.lower()
  
  /usr/local/lib/python3.6/dist-packages/keras_preprocessing/text.py in &lt;listcomp&gt;(.0)
      213                 if self.lower:
      214                     if isinstance(text, list):
  --&gt; 215                         text = [text_elem.lower() for text_elem in text]
      216                     else:
      217                         text = text.lower()
  
  AttributeError: 'list' object has no attribute 'lower'</code></pre>
  </div>
  </div>
  </p>
  
  <p>This makes sense to me, because the <code>Tokenizer</code> function expects a string but gets a list. Normally, I would flatten my list structure to pass it through the <code>Tokenizer</code> function. </p>
  
  <p><strong>However, I cannot do this because my nested list structure is essential for my modeling</strong>. </p>
  
  <p>How then, can I <code>Tokenize</code> my data while preserving the list structure? I want to treat the whole thing as my corpus and get unique word integer tokens across all lists.</p>
  
  <p>It should look something like this (tokenization done by hand here so pardon if there is a typo):</p>
  
  <pre class="lang-py prettyprint-override"><code>data = 
      [[[0],
        ['1'],
        ['2, 3, 4, 5'],
        ['6, 0, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19']],
        [['20, 21, 22, 23, 3, 24, 25'],
       ['26, 27, 28, 29']],
       [['30'],
       ['31, 32, 33, 34, 35, 36, 37, 38, 39, 40'],
       ['41, 42, 43, 44, 45, 46, 47, 48, 49'],
       ['50, 51, 34, 52, 14, 35, 53, 54, 55, 56, 17, 57, 58, 59, 31'],
       ['60, 61, 62, 63, 64, 65, 12, 66, 14, 67, 68, 59, 31']]]
  </code></pre>