title: How to delete words from a dataframe column that are present in dictionary in Pandas
body: <p>An extension to  : 
  <a href="https://stackoverflow.com/questions/25346058/removing-list-of-words-from-a-string">Removing list of words from a string</a></p>
  
  <p>I have following dataframe and I want to delete frequently occuring words from df.name column:</p>
  
  <p>df :</p>
  
  <pre><code>name
  Bill Hayden
  Rock Clinton
  Bill Gates
  Vishal James
  James Cameroon
  Micky James
  Michael Clark
  Tony Waugh  
  Tom Clark
  Tom Bill
  Avinash Clinton
  Shreyas Clinton
  Ramesh Clinton
  Adam Clark
  </code></pre>
  
  <p>I'm creating a new dataframe with words and their frequency with following code :</p>
  
  <pre><code>df = pd.DataFrame(data.name.str.split(expand=True).stack().value_counts())
  df.reset_index(level=0, inplace=True)
  df.columns = ['word', 'freq']
  df = df[df['freq'] &gt;= 3]
  </code></pre>
  
  <p>which will result in</p>
  
  <p>df2 : </p>
  
  <pre><code>word    freq
  Clinton 4
  Bill    3
  James   3
  Clark   3
  </code></pre>
  
  <p>Then I'm converting it into a dictionary with following code snippet :</p>
  
  <pre><code>    d = dict(zip(df['word'], df['freq']))
  </code></pre>
  
  <p>Now if I've to remove words from df.name that are in d(which is dictionary, with word : freq), I'm using following code snippet :       </p>
  
  <pre><code>def check_thresh_word(merc,d):
      m = merc.split(' ')
      for i in range(len(m)):
              if m[i] in d.keys():
                  return False
      else:
          return True
  
  def rm_freq_occurences(merc,d):
      if check_thresh_word(merc,d) == False:
          nwords = merc.split(' ')
          rwords = [word for word in nwords if word not in d.keys()]
          m = ' '.join(rwords)
      else:
          m=merc
      return m
  
  df['new_name'] = df['name'].apply(lambda x: rm_freq_occurences(x,d))
  </code></pre>
  
  <p>But in actual my dataframe(df) contains nearly 240k rows and i've to use threshold(thresh=3 in above sample) greater than 100.
  So above code takes lots of time to run because of complex search.
  Is there any effiecient way to make it faster??</p>
  
  <p>Following is a desired output :</p>
  
  <pre><code>name
  Hayden
  Rock
  Gates
  Vishal
  Cameroon
  Micky
  Michael
  Tony Waugh
  Tom
  Tommy
  Avinash
  Shreyas
  Ramesh
  Adam
  </code></pre>
  
  <p>Thanks in advance!!!!!!!</p>