title: How do I get the bag-of-words representation of document content with Whoosh?
body: <p>I have an index schema like the following:</p>
  
  <pre><code>schema = Schema(
      title=TEXT(stored=True),
      content=TEXT,
      id=ID,
      topicID=NUMERIC,
  )
  </code></pre>
  
  <p>I first get documents for topic <code>t</code> using <code>searcher.documents(topicID=t)</code>.  This returns hits.  I'd like to get the bag-of-words representation of the hits' <code>content</code> field.  For instance <code>[(u'This',1),(u'is',1),(u'a',1),(u'document',1)]</code> when <code>content=u'This is a document'</code>.</p>
  
  <p>If there is a way to get the bag-of-words representation (or TF-IDF) more generally in Whoosh - perhaps of documents rather than hits - that is acceptable as well.</p>
  
  <p>EDIT: I'd like a solution that preprocesses the bag-of-words/TF-IDF on indexing, and then getting the representation is a one-liner function or variable.  Instead of doing the processing live, each time I want the representation.</p>