title: Efficient way to create term density matrix from pandas DataFrame
body: <p>I am trying to create a term density matrix from a pandas dataframe, so I can rate terms appearing in the dataframe. I also want to be able to keep the 'spatial' aspect of my data (see comment at the end of post for an example of what I mean).</p>
  
  <p>I am new to pandas and NLTK, so I expect my problem to be soluble with some existing tools. </p>
  
  <p>I have a dataframe which contains two columns of interest: say 'title' and 'page'</p>
  
  <pre><code>    import pandas as pd
      import re
  
      df = pd.DataFrame({'title':['Delicious boiled egg','Fried egg ','Split orange','Something else'], 'page':[1, 2, 3, 4]})
      df.head()
  
         page                 title
      0     1  Delicious boiled egg
      1     2            Fried egg 
      2     3          Split orange
      3     4        Something else
  </code></pre>
  
  <p>My goal is to clean up the text, and pass terms of interest to a TDM dataframe. I use two functions to help me clean up the strings</p>
  
  <pre><code>    import nltk.classify
      from nltk.tokenize import wordpunct_tokenize
      from nltk.corpus import stopwords
      import string   
  
      def remove_punct(strin):
          '''
          returns a string with the punctuation marks removed, and all lower case letters
          input: strin, an ascii string. convert using strin.encode('ascii','ignore') if it is unicode 
          '''
          return strin.translate(string.maketrans("",""), string.punctuation).lower()
  
      sw = stopwords.words('english')
  
      def tok_cln(strin):
          '''
          tokenizes string and removes stopwords
          '''
          return set(nltk.wordpunct_tokenize(strin)).difference(sw)
  </code></pre>
  
  <p>And one function which does the dataframe manipulation</p>
  
  <pre><code>    def df2tdm(df,titleColumn,placementColumn,newPlacementColumn):
          '''
          takes in a DataFrame with at least two columns, and returns a dataframe with the term density matrix
          of the words appearing in the titleColumn
          Inputs: df, a DataFrame containing titleColumn, placementColumn among others
          Outputs: tdm_df, a DataFrame containing newPlacementColumn and columns with all the terms in df[titleColumn]
          '''
          tdm_df = pd.DataFrame(index=df.index, columns=[newPlacementColumn])
          tdm_df = tdm_df.fillna(0)
          for idx in df.index:
              for word in tok_cln( remove_punct(df[titleColumn][idx].encode('ascii','ignore')) ):
                  if word not in tdm_df.columns:
                      newcol = pd.DataFrame(index = df.index, columns = [word])
                      tdm_df = tdm_df.join(newcol)
          tdm_df[newPlacementColumn][idx] = df[placementColumn][idx]
          tdm_df[word][idx] = 1
          return tdm_df.fillna(0,inplace = False)
  
      tdm_df = df2tdm(df,'title','page','pub_page')
      tdm_df.head()
  </code></pre>
  
  <p>This returns</p>
  
  <pre><code>      pub_page boiled egg delicious fried orange split something else
      0        1      1   1         1     0      0     0         0    0
      1        2      0   1         0     1      0     0         0    0
      2        3      0   0         0     0      1     1         0    0
      3        4      0   0         0     0      0     0         1    1
  </code></pre>
  
  <p>But it is painfully slow when parsing large sets (output of hundred thousands of rows, thousands of columns). My two questions:</p>
  
  <p>Can I speed up this implementation?</p>
  
  <p>Is there some other tool I could use to get this done? </p>
  
  <p>I want to be able to keep the 'spatial' aspect of my data, for example if 'egg' appears very often in pages 1-10 and then reappears often in pages 500-520, I want to know that.</p>