title: Processing Text for Classification with Keras
body: <p>I'm trying to train a basic text classification NN using Keras. I download 12,500 pos and 12,500 negative movie reviews from a website. I'm having trouble processing the data into something Keras can use however. </p>
  
  <p>First, I open the 25000 text files and store each file into an array. I then run each array (one positive and one negative) through this function:</p>
  
  <pre><code>def process_for_model(textArray):
      '''
       Given a 2D array of the form:
       [[fileLines1],[fileLines2]...[fileLinesN]]
       converts the text into integers
      '''
      result = []
      for file_ in textArray:
          inner = []
          for line in file_:
              length = len(set(text_to_word_sequence(line)))
              inner.append(hashing_trick(line,round(length*1.3),hash_function='md5'))
          result.append(inner)
  
      return result
  </code></pre>
  
  <p>With the purpose of converting the words into numbers to get them close to something a Keras model can use.</p>
  
  <p>I then append the converted numbers into a single array, along with appending a 0 or 1 to another array as labels:</p>
  
  <pre><code>training_labels = []
  train_batches = []
  for i in range(len(positive_encoded)):
      train_batches.append(positive_encoded[i])
      training_labels.append([0])
  for i in range(len(negative_encoded)):
      train_batches.append(negative_encoded[i])
      training_labels.append([1])
  </code></pre>
  
  <p>And finally I convert each array to a np array:</p>
  
  <pre><code>train_batches = array(train_batches)
  training_labels = array(training_labels)
  </code></pre>
  
  <p>However, I'm not really sure where to go from here. Each review is, I believe, 168 words. I don't know how to create an appropriate model for this data or how to properly scale all the numbers to be between 0 and 1 using sklearn.</p>
  
  <p>The things I am most confused on are: how many layers should I have, how many neurons each layer should have, and how many input dimensions should I have for the first layer.</p>
  
  <p>Should I be taking another approach entirely?</p>