title: Why does using multiprocessing with pandas apply lead to such a dramatic speedup?
body: <p>Suppose I have a pandas dataframe and a function I'd like to apply to each row.  I can call <code>df.apply(apply_fn, axis=1)</code>, which should take time linear in the size of <code>df</code>.  Or I can split <code>df</code> and use <code>pool.map</code> to call my function on each piece, and then concatenate the results.</p>
  
  <p>I was expecting the speedup factor from using <code>pool.map</code> to be roughly equal to the number of processes in the pool (new_execution_time = original_execution_time/N if using N processors -- and that's assuming zero overhead).</p>
  
  <p>Instead, in this toy example, time falls to around 2% (0.005272 / 0.230757) when using 4 processors.  I was expecting 25% at best.  What is going on and what am I not understanding?</p>
  
  <pre><code>import numpy as np
  from multiprocessing import Pool
  import pandas as pd
  import pdb
  import time
  
  n = 1000
  variables = {"hello":np.arange(n), "there":np.random.randn(n)}
  df = pd.DataFrame(variables)
  
  def apply_fn(series):
      return pd.Series({"col_5":5, "col_88":88,
                        "sum_hello_there":series["hello"] + series["there"]})
  
  def call_apply_fn(df):
      return df.apply(apply_fn, axis=1)
  
  n_processes = 4  # My machine has 4 CPUs
  pool = Pool(processes=n_processes)
  
  t0 = time.process_time()
  new_df = df.apply(apply_fn, axis=1)
  t1 = time.process_time()
  df_split = np.array_split(df, n_processes)
  pool_results = pool.map(call_apply_fn, df_split)
  new_df2 = pd.concat(pool_results)
  t2 = time.process_time()
  new_df3 = df.apply(apply_fn, axis=1)  # Try df.apply a second time
  t3 = time.process_time()
  
  print("identical results: %s" % np.all(np.isclose(new_df, new_df2)))  # True
  print("t1 - t0 = %f" % (t1 - t0))  # I got 0.230757
  print("t2 - t1 = %f" % (t2 - t1))  # I got 0.005272
  print("t3 - t2 = %f" % (t3 - t2))  # I got 0.229413
  </code></pre>
  
  <p>I saved the code above and ran it using <code>python3 my_filename.py</code>.</p>
  
  <p>PS  I realize that in this toy example <code>new_df</code> can be created in a much more straightforward way, without using apply.  I'm interested in applying similar code with a more complex <code>apply_fn</code> that doesn't just add columns.</p>