title: How to iterate over consecutive chunks of Pandas dataframe efficiently
body: <p>I have a large dataframe (several million rows).</p>
  
  <p>I want to be able to do a groupby operation on it, but just grouping by arbitrary consecutive (preferably equal-sized) subsets of rows, rather than using any particular property of the individual rows to decide which group they go to.</p>
  
  <p>The use case: I want to apply a function to each row via a parallel map in IPython. It doesn't matter which rows go to which back-end engine, as the function calculates a result based on one row at a time. (Conceptually at least; in reality it's vectorized.)</p>
  
  <p>I've come up with something like this:</p>
  
  <pre><code># Generate a number from 0-9 for each row, indicating which tenth of the DF it belongs to
  max_idx = dataframe.index.max()
  tenths = ((10 * dataframe.index) / (1 + max_idx)).astype(np.uint32)
  
  # Use this value to perform a groupby, yielding 10 consecutive chunks
  groups = [g[1] for g in dataframe.groupby(tenths)]
  
  # Process chunks in parallel
  results = dview.map_sync(my_function, groups)
  </code></pre>
  
  <p>But this seems very long-winded, and doesn't guarantee equal sized chunks. Especially if the index is sparse or non-integer or whatever.</p>
  
  <p>Any suggestions for a better way?</p>
  
  <p>Thanks!</p>