title: Java Lucene Stop Words Filter
body: <p>I have about 500 sentences in which I would like to compile a set of ngrams. I am having trouble removing the stop words. I tried adding the lucene StandardFilter and StopFilter but I still have the same problem. Here is my code:</p>
  
  <pre><code>for(String curS: Sentences)
  {
            reader = new StringReader(curS);
            tokenizer = new StandardTokenizer(Version.LUCENE_36, reader);
            tokenizer = new StandardFilter(Version.LUCENE_36, tokenizer);
            tokenizer = new StopFilter(Version.LUCENE_36, tokenizer, stopWords);
            tokenizer = new ShingleFilter(tokenizer, 2, 3);
            charTermAttribute = tokenizer.addAttribute(CharTermAttribute.class);
  
      while(tokenizer.incrementToken())
      {
          curNGram = charTermAttribute.toString().toString();
          nGrams.add(curNGram);                   //store each token into an ArrayList
      }
  }
  </code></pre>
  
  <p>For example, the first phrase I am testing is: "For every person that listens to". In this example curNGram is set to "For" which is a stop word in my list stopWords. Also, in this example "every" is a stop word and so "person" should be the first ngram.</p>
  
  <ol>
  <li>Why are stop words being added to my list when I am using the StopFiler?</li>
  </ol>
  
  <p>All help is appreciated!</p>