title: Creating a TfidfVectorizer over a text column of huge pandas dataframe
body: <p>I need to get matrix of TF-IDF features from the text stored in columns of a  huge <a href="https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.html" rel="nofollow noreferrer">dataframe</a>, loaded from a CSV file (which cannot fit in memory). I am trying to iterate over dataframe using chunks but it is returning generator objects which is not an expected variable type for the method <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer" rel="nofollow noreferrer">TfidfVectorizer</a>. I guess I am doing something wrong while writing a generator method <code>ChunkIterator</code>shown below.</p>
  
  <pre><code>import pandas as pd
  from sklearn.feature_extraction.text import TfidfVectorizer
  
  
  #Will work only for small Dataset
  csvfilename = 'data_elements.csv'
  df = pd.read_csv(csvfilename)
  vectorizer = TfidfVectorizer()
  corpus  = df['text_column'].values
  vectorizer.fit_transform(corpus)
  print(vectorizer.get_feature_names())
  
  
  
  #Trying to use a generator to parse over a huge dataframe
  def ChunkIterator(filename):
      for chunk in pd.read_csv(csvfilename, chunksize=1):
         yield chunk['text_column'].values
  
  corpus  = ChunkIterator(csvfilename)
  vectorizer.fit_transform(corpus)
  print(vectorizer.get_feature_names())
  </code></pre>
  
  <p>Can anybody please advise how to modify the <code>ChunkIterator</code> method above, or any other approach using <a href="https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.html" rel="nofollow noreferrer">dataframe</a>. I would like to avoid creating separate text files for each row in the <a href="https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.html" rel="nofollow noreferrer">dataframe</a>. Following is some dummy csv file data for recreating the scenario.</p>
  
  <pre><code>id,text_column,tags
  001, This is the first document .,['sports','entertainment']
  002, This document is the second document .,"['politics', 'asia']"
  003, And this is the third one .,['europe','nato']
  004, Is this the first document ?,"['sports', 'soccer']"
  </code></pre>