title: scikit-learn: fitting data into chunks vs fitting it all at once
body: <p>I am using scikit-learn to build a classifier, which works on (somewhat large) text files. I need a simple bag-of-words features at the moment, therefore I tried using TfidfVectorizer/HashingVectorizer/CountVectorizer to obtain the feature vectors. </p>
  
  <p>However, processing the entire train data at once to obtain the feature vectors results in memory error in numpy/scipy (depending on which vectorizer I use). So my question is: </p>
  
  <p><strong>When extracting text features from the raw text: if I fit the data to the vectorizer in chunks, will that be the same as fitting the entire data at once?</strong></p>
  
  <p>To illustrate this with code, is the following:</p>
  
  <pre><code>vectoriser = CountVectorizer() # or TfidfVectorizer/HashingVectorizer
  train_vectors = vectoriser.fit_transform(train_data)
  </code></pre>
  
  <p>different from the following:</p>
  
  <pre><code>vectoriser = CountVectorizer() # or TfidfVectorizer/HashingVectorizer
  
  
  start = 0
  while start &lt; len(train_data):
      vectoriser.fit(train_data[start:(start+500)])
      start += 500
  
  train_vectors = vectoriser.transform(train_data)
  </code></pre>
  
  <p>Thanks in advance and sorry if this question is completely retarded. </p>