title: How to get a Token from a Lucene TokenStream?
body: <p>I'm trying to use Apache Lucene for tokenizing, and I am baffled at the process to obtain Tokens from a <code>TokenStream</code>.</p>
  
  <p>The worst part is that I'm looking at the comments in the JavaDocs that address my question.</p>
  
  <p><a href="http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/analysis/TokenStream.html#incrementToken%28%29" rel="noreferrer">http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/analysis/TokenStream.html#incrementToken%28%29</a></p>
  
  <p>Somehow, an <code>AttributeSource</code> is supposed to be used, rather than <code>Token</code>s.  I'm totally at a loss.  </p>
  
  <p>Can anyone explain how to get token-like information from a TokenStream?</p>