title: Creating ngrams from scikit learn and count vectorizer throws Memory Error
body: <p>I am building <strong>ngrams</strong> from multiple text documents using <strong>scikit-learn</strong>. I need to build <strong>document-frequency</strong> using <strong>countVectorizer</strong>. </p>
  
  <p><strong>Example</strong> : </p>
  
  <pre><code>document1 = "john is a nice guy"
  
  document2 = "person can be a guy"
  </code></pre>
  
  <p>So, <strong>document-frequency</strong> will be</p>
  
  <pre><code>{'be': 1,
   'can': 1,
   'guy': 2,
   'is': 1,
   'john': 1,
   'nice': 1,
   'person': 1}
  </code></pre>
  
  <p>Here documents are just strings but when I tried with <strong>huge amount of data</strong>. It throws <strong>MEMORY ERROR.</strong> </p>
  
  <p><strong>Code :</strong></p>
  
  <pre><code>import numpy as np
  from sklearn.feature_extraction.text import CountVectorizer
  document = [Huge amount of data around 7MB] # ['john is a guy', 'person guy']
  vectorizer = CountVectorizer(ngram_range=(1, 5))
  X = vectorizer.fit_transform(document).todense()
  tranformer = vectorizer.transform(document).todense()
  matrix_terms = np.array(vectorizer.get_feature_names())
  lst_freq =  map(sum,zip(*tranformer.A))          
  matrix_freq = np.array(lst_freq)
  final_matrix = np.array([matrix_terms,matrix_freq])
  </code></pre>
  
  <p><strong>ERROR :</strong></p>
  
  <pre><code>Traceback (most recent call last):
    File "demo1.py", line 13, in build_ngrams_matrix
      X = vectorizer.fit_transform(document).todense()
    File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", line 605, in todense
      return np.asmatrix(self.toarray(order=order, out=out))
    File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py", line 901, in toarray
      return self.tocoo(copy=False).toarray(order=order, out=out)
    File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/coo.py", line 269, in toarray
      B = self._process_toarray_args(order, out)
    File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", line 789, in _process_toarray_args
      return np.zeros(self.shape, dtype=self.dtype, order=order)
  MemoryError
  </code></pre>