title: Slow speed while parallelizing operation on pandas dataframe
body: <p>I have a dataframe which I perform some operation on and print out. To do this, I have to iterate through each row.</p>
  
  <pre><code>for count, row in final_df.iterrows():
      x = row['param_a']
      y = row['param_b']
      # Perform operation
      # Write to output file
  </code></pre>
  
  <p>I decided to parallelize this using the python multiprocessing module</p>
  
  <pre><code>def write_site_files(row):
      x = row['param_a']
      y = row['param_b']
      # Perform operation
      # Write to output file
  
  pkg_num = 0
  total_runs = final_df.shape[0] # Total number of rows in final_df
  threads = []
  
  import multiprocessing
  
  while pkg_num &lt; total_runs or len(threads):
      if(len(threads) &lt; num_proc and pkg_num &lt; total_runs):
          print pkg_num, total_runs
          t = multiprocessing.Process(target=write_site_files,args=[final_df.iloc[pkg_num],pkg_num])
          pkg_num = pkg_num + 1
          t.start()
          threads.append(t)
      else:
          for thread in threads:
              if not thread.is_alive():
                 threads.remove(thread)
  </code></pre>
  
  <p>However, the latter (parallelized) method is way slower than the simple iteration based approach. Is there anything I am missing?</p>
  
  <p>thanks!</p>