title: dask.multiprocessing or pandas + multiprocessing.pool: what's the difference?
body: <p>I'm developing a model for financial purpose. I have the entire S&amp;P500 components inside a folder, stored as many .hdf files. Each .hdf file has its own multi-index (year-week-minute).</p>
  <p><strong>An example of the sequential code (non parallelized):</strong></p>
  <pre><code>import os
  from classAsset import Asset
  
  
  def model(current_period, previous_perdiod):
      # do stuff on the current period, based on stats derived from previous_period
      return results
  
  if __name__ == '__main__':
      for hdf_file in os.listdir('data_path'):
          asset = Asset(hdf_file)
          for year in asset.data.index.get_level_values(0).unique().values:
              for week in asset.data.loc[year].index.get_level_values(0).unique().values:
  
                  previous_period = asset.data.loc[(start):(end)].Open.values  # start and end are defined in another function
                  current_period = asset.data.loc[year, week].Open.values
  
                  model(current_period, previous_period)
  </code></pre>
  <p>To speed up the process, I'm using multiprocessing.pool to run the same algorithm on multiple .hdf files at the same time, so I'm quite satisfied with the processing speed (I have a 4c/8t CPU). But now I discovered Dask.</p>
  <p>In <a href="http://dask.pydata.org/en/latest/dataframe-overview.html#scope" rel="noreferrer">Dask documentation 'DataFrame Overview'</a> they indicate:</p>
  <p><strong>Trivially parallelizable operations (fast)</strong>:</p>
  <ul>
  <li>Elementwise operations: df.x + df.y, df * df</li>
  <li>Row-wise selections: df[df.x &gt; 0]</li>
  <li>Loc: df.loc[4.0:10.5]  (<em>this is what interests me the most</em>)</li>
  </ul>
  <p>Also, in <a href="http://dask.pydata.org/en/latest/use-cases.html#embarrassingly-parallel-computation" rel="noreferrer">Dask documentation 'Use Cases'</a> they indicate:</p>
  <blockquote>
  <p><em>A programmer has a function that they want to run many times on different inputs. Their function and inputs might use arrays or dataframes internally, but conceptually their problem isn’t a single large array or dataframe.</em></p>
  <p><em>They want to run these functions in parallel on their laptop while they prototype but they also intend to eventually use an in-house cluster. They wrap their function in dask.delayed and let the appropriate dask scheduler parallelize and load balance the work.</em></p>
  </blockquote>
  <p>So I'm sure I'm missing something, or probably more than just something. What's the difference between processing many single pandas dataframes with multiprocessing.pool and dask.multiprocessing?</p>
  <p>Do you think I should use Dask for my specific case? Thank you guys.</p>