title: How is term frequency calculated in scikit-learn CountVectorizer
body: <p>I do not understand how CountVectorizer calculates the term frequency. I need to know this so that I can make a sensible choice for the <code>max_df</code> parameter when filtering out terms from a corpus. Here is example code:</p>
  
  <pre><code>    import pandas as pd
      import numpy as np
      from sklearn.feature_extraction.text import CountVectorizer
  
      vectorizer = CountVectorizer(min_df = 1, max_df = 0.9)
      X = vectorizer.fit_transform(['afr bdf dssd','afr bdf c','afr'])
      word_freq_df = pd.DataFrame({'term': vectorizer.get_feature_names(), 'occurrences':np.asarray(X.sum(axis=0)).ravel().tolist()})
      word_freq_df['frequency'] = word_freq_df['occurrences']/np.sum(word_freq_df['occurrences'])
      print word_freq_df.sort('occurrences',ascending = False).head()
  
         occurrences  term  frequency
      0            3   afr   0.500000
      1            2   bdf   0.333333
      2            1  dssd   0.166667
  </code></pre>
  
  <p>It seems that 'afr' appears in half of the terms in my corpus, as I expect by looking at the corpus. However, when I set <code>max_df  = 0.8</code> in <code>CountVectorizer</code>, the term 'afr' is filtered out of my corpus. Playing around, I find that with the coprus in my example, CountVectorizer assigns a frequency of ~0.833 to 'afr'. Could someone provide a formula on how the term frequency which enterts <code>max_df</code> is calculated?</p>
  
  <p>Thanks</p>