title: Word importance in lucene index
body: <p>hmmm, i need to get how <strong>important is the word in entire document collection</strong> that is <strong>indexed in the lucene index</strong>.  I need to extract some "representable words", lets say concepts that are common and can be representable to whole collection. Or collection "keywords". I did the fulltext indexing and the only field i am using are text contents, because titles of the documents are mostly not representable(numbers, codes etc....)</p>
  
  <p>EDIT:
  I am reading the index which contains maybe 60 documents....</p>
  
  <pre><code> int numDocs = fReader.numDocs();
   while(termEnum.next())
      {
          Term term = termEnum.term();
          double df = fReader.docFreq(term); 
  
         TermDocs termDocs = indexReader.termDocs(term);
  
      //HERE is what i mean when i say tfidf is per document,
  
               while(termDocs.next())
              {
                 double tf = termDocs.freq();
                 // Calculate tfidf.......
              }
  
              termDocs.close();
  </code></pre>
  
  <p>}</p>
  
  <p>So, I will get tfidf of this term, but for every document that we loop through. And I do not need these results: </p>
  
  <p>tfidf(term1, doc1);</p>
  
  <p>tfidf(term1, doc2);</p>
  
  <p>tfidf(term1, doc3);
  ...........and so on.<br>
  I need some measure of importance of this term in the collection. By intuition, it would be something like "if term "term1" had good tfidf in 5 documents then it is important"</p>
  
  <p>But ofcourse, something smarter :)</p>
  
  <p>Thank you!!!</p>