title: Fastest way to write HDF5 files with Python?
body: <p>Given a large (10s of GB) CSV file of mixed text/numbers, what is the fastest way to create an HDF5 file with the same content, while keeping the memory usage reasonable?</p>
  
  <p>I'd like to use the <code>h5py</code> module if possible.</p>
  
  <p>In the toy example below, I've found an incredibly slow and incredibly fast way to write data to HDF5. Would it be best practice to write to HDF5 in chunks of 10,000 rows or so? Or is there a better way to write a massive amount of data to such a file?</p>
  
  <pre><code>import h5py
  
  n = 10000000
  f = h5py.File('foo.h5','w')
  dset = f.create_dataset('int',(n,),'i')
  
  # this is terribly slow
  for i in xrange(n):
    dset[i] = i
  
  # instantaneous
  dset[...] = 42
  </code></pre>