title: removing words that appear more than x% in a corpus Python
body: <p>I am dealing with a large corpus in the form of a list of tokens/words. The corpus contains ~1900,000 words, I have run a code to get the most frequent words and now the corpus has 140,000 word.</p>
  
  <p>I would like to remove the words that appear more than 95% of the document and less than 5% </p>
  
  <p>sample of the corpus</p>
  
  <pre><code>['problems', 'guess', 'sleep', 'holy']
  </code></pre>
  
  <p>First I found the most frequent words</p>
  
  <pre><code>from nltk.probability import FreqDist
  
  corpus_frequency = FreqDist(corpus)
  
  corpus_commom=corpus_frequency.most_common()
  </code></pre>
  
  <p>Then, I applied this for loop to find the list of words that appear more than 95%</p>
  
  <pre><code>most_frequent=[mytuple for mytuple in corpus_commom if mytuple[1]&lt;len(corpus*95)/100]
  </code></pre>
  
  <p>But this code takes very long time to run, and it does not return any output.</p>
  
  <p>I also tried to follow some answers I found and apply CountVectorizer but I get an error message </p>
  
  <pre><code>from sklearn.feature_extraction.text import CountVectorizer
  cv = CountVectorizer(min_df=0.05, max_df=0.95, lowercase=True) 
  X = cv.fit_transform(corpus)
  </code></pre>
  
  <p>The error message</p>
  
  <pre><code>ValueError: After pruning, no terms remain. Try a lower min_df or a higher max_df.
  </code></pre>
  
  <p><strong>Update</strong></p>
  
  <p>The CountVectorizer works if I do this </p>
  
  <pre><code>cv = CountVectorizer(corpus,max_df=0.95)
  count_vector=cv.fit_transform(corpus)
  </code></pre>
  
  <p>But it returns a list of numbers. I would like the output to be my corpus(list of words) filtered. </p>
  
  <p>Can anyone give me a tip on how to achieve this? Thanks </p>