title: How to process rows of a pandas DataFrame in parallel in Python
body: <h1>Exemplary dummy example:</h1>
  <p>I have a DataFrame <code>df</code>:</p>
  <pre><code>&gt; df
  
         para0  para1   para2
  0  17.439020   True    high
  1  19.757758   True    high
  2  12.434424   True  medium
  3  14.789654   True     low
  4  14.131464  False    high
  5   9.900233   True    high
  6  10.977869  False     low
  7   8.004251   True  medium
  8  11.468420  False     low
  9  12.764453  False    high
  </code></pre>
  <p>in which each row consists of a collection of parameters for a function
  <code>foobar</code>:</p>
  <pre><code>def foobar(r):
      &quot;&quot;&quot; r is a row of df, does something, and it takes a long time&quot;&quot;&quot;
      if r.para1:
          x = r.para2
      else:
          x = 'low'
      return int(r.para0), (r.Index+13)%3 == 0, x
  </code></pre>
  <p>I would like to apply <code>foobar</code> to each row of <code>df</code>, collect its
  results, and store these together with their respected parameters in a,
  well, DataFrame.</p>
  <p>My (current) solution:</p>
  <pre><code>df['count'] = 0
  df['valid'] = False
  df['outpt'] = ''
  
  def wrapper(r, df):
      c, v, o = foobar(r)
      df.ix[r.Index,'count'] = c
      df.ix[r.Index,'valid'] = v
      df.ix[r.Index,'outpt'] = o
  
  for r in df.itertuples():
      wrapper(r, df)
  </code></pre>
  <p>This yields:</p>
  <pre><code>&gt; df
         para0  para1   para2  count  valid   outpt
  0  17.439020   True    high   17.0  False    high
  1  19.757758   True    high   19.0  False    high
  2  12.434424   True  medium   12.0   True  medium
  3  14.789654   True     low   14.0  False     low
  4  14.131464  False    high   14.0  False     low
  5   9.900233   True    high    9.0   True    high
  6  10.977869  False     low   10.0  False     low
  7   8.004251   True  medium    8.0  False  medium
  8  11.468420  False     low   11.0   True     low
  9  12.764453  False    high   12.0  False     low
  </code></pre>
  <h1>Here is my question:</h1>
  <p>In real life, the function <code>foobar</code> is
  computational expensive and takes approximately 20-30 min to run, <code>df</code>
  has typically between 100-2000 rows.  I have access to a machine with
  eight cores, and as <code>foobar</code> only depends on the current processed row
  and neither on anything else, it should be trivial to run these
  computations in parallel.</p>
  <p>It would also be nice that, when something goes wrong (say, if someone
  accidentally turns off the machine), it would not be necessary to start
  everything from the beginning, i.e., to skip rows which have already
  been processed.</p>
  <p>How can I do this?</p>
  <hr />
  <p>My try on <code>multiprocessing</code> unfortunately failed:</p>
  <pre><code>from multiprocessing import Pool
  
  pool = Pool(3)
  results = []
  
  for r in df.itertuples():
      results += [pool.apply_async(wrapper, r, df)]
  </code></pre>
  <p>With:</p>
  <pre><code>&gt; results[0].get()
  …
  /usr/lib/python3.5/multiprocessing/reduction.py in dumps(cls, obj, protocol)
       48     def dumps(cls, obj, protocol=None):
       49         buf = io.BytesIO()
  ---&gt; 50         cls(buf, protocol).dump(obj)
       51         return buf.getbuffer()
       52
  
  PicklingError: Can't pickle &lt;class 'pandas.core.frame.Pandas'&gt;: attribute lookup Pandas on pandas.core.frame failed
  </code></pre>
  <p>Here is how I created the toy DataFrame:</p>
  <pre><code>import pandas as pd
  import numpy as np
  
  df = pd.DataFrame({
      'para0' : pd.Series(
          np.random.gamma(12,size=10),
          dtype=np.float),
      'para1' : pd.Series(
          [(True,False)[i] for i in np.random.randint(0,2,10)],
          dtype=np.bool),
      'para2' : pd.Categorical(
          [('low','medium','high')[i] for i in np.random.randint(0,3,10)],
          ordered=True),
      })
  </code></pre>