title: Removing Stop words without tokenizing in Python
body: <p>I am trying to remove stop words from a list of strings from the exercise: <a href="https://colab.research.google.com/github/lmoroney/dlaicourse/blob/master/TensorFlow%20In%20Practice/Course%203%20-%20NLP/Course%203%20-%20Week%201%20-%20Exercise-answer.ipynb" rel="nofollow noreferrer">ipython file</a>, my solution is as follows:</p>
  <pre><code>sentences = []
  labels = []
  with open(&quot;./bbc-text.csv&quot;, 'r') as csvfile:
      reader = csv.reader(csvfile, delimiter =',')
      next(reader)
      for row in reader: 
          labels.append(row[0])
          # clean up the sentence
          sentence = row[1]
          for word in stopwords: 
              if word in sentence: 
                  sentence = sentence.replace(&quot; &quot;+ word + &quot; &quot;, &quot;&quot;)
          sentences.append(sentence)
  
  print(len(sentences))
  </code></pre>
  <p>But when I tokenize the words, the word index is 131530 which is much larger that it is expected:</p>
  <pre><code>tokenizer = Tokenizer(oov_token=&quot;&lt;OOV&gt;&quot;)
  tokenizer.fit_on_texts(sentences)
  word_index = tokenizer.word_index
  print(len(word_index))
  # Expected output
  # 29714 
  </code></pre>
  <p>Here is the solution provided by the instructor:</p>
  <pre><code>sentences = []
  labels = []
  with open(&quot;./bbc-text.csv&quot;, 'r') as csvfile:
      reader = csv.reader(csvfile, delimiter=',')
      next(reader)
      for row in reader:
          labels.append(row[0])
          sentence = row[1]
          for word in stopwords:
              token = &quot; &quot; + word + &quot; &quot;
              sentence = sentence.replace(token, &quot; &quot;)
              sentence = sentence.replace(&quot;  &quot;, &quot; &quot;)
          sentences.append(sentence)
  </code></pre>
  <p>What am I doing wrong please?</p>
  <p>Thank you,
  CS</p>