title: ShingleFilter in Lucene 4.6 adds words to bigrams
body: <p>I am puzzled by the strange behavior of ShingleFilter in Lucene 4.6.   What I would like to do is extract all possible bigrams from a sentence.  So if the sentence is "this is a dog", I want "this is", "is a ", "a dog".  </p>
  
  <p>What I see instead is:</p>
  
  <p>"this this is"</p>
  
  <p>"this is is"</p>
  
  <p>"is is a"</p>
  
  <p>"is a a"</p>
  
  <p>"a a dog"</p>
  
  <p>"a dog dog"</p>
  
  <p>So somehow it replicates words and makes 3-grams instead of bigrams.</p>
  
  <p>Here is my Java code:</p>
  
  <pre><code>Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_46);
  ShingleAnalyzerWrapper shingleAnalyzer = new ShingleAnalyzerWrapper(analyzer,2,2);
  
  String theSentence = "this is a dog";
  StringReader reader = new StringReader(theSentence);
  TokenStream tokenStream = shingleAnalyzer.tokenStream("content", reader);
  ShingleFilter theFilter = new ShingleFilter(tokenStream);
  theFilter.setOutputUnigrams(false);
  
  CharTermAttribute charTermAttribute = theFilter.addAttribute(CharTermAttribute.class);                                                                                   
  theFilter.reset();
  
   while (theFilter.incrementToken()) {                                                 
              System.out.println(charTermAttribute.toString());                                  
   }
  
   theFilter.end();
   theFilter.close();
  </code></pre>
  
  <p>Could someone please help me modify it so it actually outputs bigrams and does not replicate words?  Thank you!</p>
  
  <p>Natalia </p>