title: Why do pandas and dask perform better when importing from CSV compared to HDF5?
body: <p>I am working with a system that currently operates with large (>5GB) .csv files. To increase performance, I am testing (A) different methods to create dataframes from disk (pandas VS <a href="http://pythondata.com/dask-large-csv-python/" rel="nofollow noreferrer">dask</a>) as well as (B) different ways to store results to disk (.csv VS <a href="https://dzone.com/articles/quick-hdf5-pandas" rel="nofollow noreferrer">hdf5</a> files).</p>
  
  <p>In order to benchmark performance, I did the following:</p>
  
  <pre><code>def dask_read_from_hdf():
      results_dd_hdf = dd.read_hdf('store.h5', key='period1', columns = ['Security'])
      analyzed_stocks_dd_hdf =  results_dd_hdf.Security.unique()
      hdf.close()
  
  def pandas_read_from_hdf():
      results_pd_hdf = pd.read_hdf('store.h5', key='period1', columns = ['Security'])
      analyzed_stocks_pd_hdf =  results_pd_hdf.Security.unique()
      hdf.close()
  
  def dask_read_from_csv():
      results_dd_csv = dd.read_csv(results_path, sep = ",", usecols = [0], header = 1, names = ["Security"])
      analyzed_stocks_dd_csv =  results_dd_csv.Security.unique()
  
  def pandas_read_from_csv():
      results_pd_csv = pd.read_csv(results_path, sep = ",", usecols = [0], header = 1, names = ["Security"])
      analyzed_stocks_pd_csv =  results_pd_csv.Security.unique()
  
  print "dask hdf performance"
  %timeit dask_read_from_hdf()
  gc.collect()
  print""
  print "pandas hdf performance"
  %timeit pandas_read_from_hdf()
  gc.collect()
  print""
  print "dask csv performance"
  %timeit dask_read_from_csv()
  gc.collect()
  print""
  print "pandas csv performance"
  %timeit pandas_read_from_csv()
  gc.collect()
  </code></pre>
  
  <p>My findings are:</p>
  
  <pre><code>dask hdf performance
  10 loops, best of 3: 133 ms per loop
  
  pandas hdf performance
  1 loop, best of 3: 1.42 s per loop
  
  dask csv performance
  1 loop, best of 3: 7.88 ms per loop
  
  pandas csv performance
  1 loop, best of 3: 827 ms per loop
  </code></pre>
  
  <p>When hdf5 storage can be accessed faster than .csv, and when dask creates dataframes faster than pandas, why is dask from hdf5 slower than dask from csv? Am I doing something wrong?</p>
  
  <p><strong>When does it make sense for performance to create dask dataframes from HDF5 storage objects?</strong></p>