title: appending rows with pandas' to_hdf multiples H5 file size?
body: <p>I have an <code>HDF5</code> with about 13,000 rows × 5 columns, these rows were appended over time to the same file with <code>DF.to_hdf(Filename, 'df', append=True, format='table')</code> and here's the size:</p>
  <pre><code>-rw-r--r--  1 omnom  omnom   807M Mar 10 15:55 Final_all_result.h5
  </code></pre>
  <p>Recently I received a <code>ValueError</code> because the data I was trying to append to one of the columns is longer than the declared column size (2000, with <code>min_itemsize</code>).</p>
  <p>So I loaded all rows to memory and dump these into a new <code>HDF</code> file <strong>at one go</strong> with:</p>
  <pre><code>DF.to_hdf(newFilename, \
                  'df', \
                  mode='a', \
                  data_columns=['Code', 'ID', 'Category', 'Title', 'Content'], \
                  format='table', \
                  min_itemsize={'index': 24, \
                                  'Code': 8, \
                                  'ID': 32, \
                                  'Category': 24, \
                                  'Title': 192, \
                                  'Content':5000 \
                                  } \
                  )
  </code></pre>
  <p>I was really surprised that the new file size is about 1/10 of the original file:</p>
  <pre><code>-rw-r--r--  1 omnom  omnom    70M Mar 10 16:01 Final_all_result_5000.h5
  </code></pre>
  <p>I double checked the number of rows in both files, they're equal.</p>
  <p>Do I append new rows the wrong way that causes the file size to multiple with every append operation? Googled and searched here but don't think this was discussed before, or maybe I searched with the wrong keywords.</p>
  <p>Any help is appreciated.</p>
  <p>UPDATE:
  I tried adding <code>min_itemsize</code> for all data columns in the append line per suggestion in this thread: <a href="https://stackoverflow.com/questions/22934996/pandas-pytables-append-performance-and-increase-in-file-size">pandas pytables append: performance and increase in file size</a>:</p>
  <pre><code>DF.to_hdf(h5AbsPath, \
                  'df', \
                  mode='a', \
                  data_columns=['Code', 'ID', 'Category', 'Title', 'Content'], \
                  min_itemsize={'index': 24, \
                                  'Code': 8, \
                                  'ID': 32, \
                                  'Category': 24, \
                                  'Title': 192, \
                                  'Content':5000 \
                                  }, \
                   append=True \
                   )
  </code></pre>
  <p>but still it doesn't reduce file size.</p>
  <p>Thanks for suggestions to add compression, both the appended and newly dumped files are not compressed per requirement.</p>