title: Solr custom filter TokenStream issue
body: <p>For indexing and querying I need to perform certain transformations as listed below. Therefore I wrote a custom filter. How can I perform the concatenation of the token and it get passed to NGramFilterFactory filter. Please tell me what needs to be improved in the code.</p>
  
  <p>This is the config from schema.xml file:</p>
  
  <pre class="lang-xml prettyprint-override"><code>&lt;tokenizer class="solr.WhitespaceTokenizerFactory"/&gt;
  
      &lt;filter class="solr.LowerCaseFilterFactory" /&gt;
      &lt;filter class="solr.TrimFilterFactory" /&gt;
      &lt;filter class="solr.TrimFilterFactory" pattern="([^a-z])" replacement="" replace="all" /&gt;
      &lt;filter class="solr.StopFilterFactory" ignoreCase="true" words="custom_stop_words.txt"/&gt;
      &lt;filter class="intuit.ripple.solr.ConcatFilterFactory" /&gt;
      &lt;filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="3" /&gt;
  </code></pre>
  
  <p>Here is an example of the use case I am trying to solve:</p>
  
  <pre><code>1. Input value: "foo Bar Baz qux"
  2. WhitespaceTokenizerFactory: "foo", "Bar", "Baz", "qux"
  3. LowerCaseFilterFactory: "foo", "bar", "baz", "qux"
  4. TrimFilterFactory, TrimFilterFactory and StopFilterFactory have nothing to do in this case.
  5. ConcatFilterFactory: "foobarbazqux". It should concatenate the tokens.
  6. NGramFilterFactory: This will generate the token.
  </code></pre>
  
  <p>Here is the <code>incrementToken()</code> method from ConcatFilter:</p>
  
  <pre class="lang-java prettyprint-override"><code>@Override
  public boolean incrementToken()  throws IOException  {
  
      StringBuilder builder = new StringBuilder();
  
      while (input.incrementToken()) {
          int len = charTermAtt.length();
          char buffer[] = charTermAtt.buffer();
          builder.append(buffer, 0, len);
          System.out.println("Tokens: " + new String(buffer, 0, len));
          clearAttributes();
          charTermAtt.setEmpty();
      }
      System.out.println("Concat tokens: " + builder.toString());
  
      charTermAtt.copyBuffer(builder.toString().toCharArray(), 0, builder.length());
      charTermAtt.setLength(builder.length());
      posIncAtt.setPositionIncrement(1);
      setOffsetAttr.setOffset(0, builder.length());
  
      input.end();
      input.close();
      return false;
  }
  </code></pre>
  
  <p>Here I am using while loop to fetch all the token and joining them together. Is there any way to fetch all the token at once without looping?</p>