 <p>Answering question 2, with pandas 0.18.0 you can do:</p> <pre><code>store = pd.HDFStore('compiled_measurements.h5') for filepath in file_iterator: raw = pd.read_csv(filepath) store.append('measurements', raw, index=False) store.create_table_index('measurements', columns=['a', 'b', 'c'], optlevel=9, kind='full') store.close() </code></pre> <p>Based on <a href="http://pandas.pydata.org/pandas-docs/stable/io.html#indexing" rel="nofollow">this</a> part of the docs.</p> <p>Depending on how much data you have, the index creation can consume enormous amounts of memory. The PyTables docs describes the values of <a href="http://www.pytables.org/usersguide/libref/structured_storage.html#column-methods" rel="nofollow">optlevel</a>.</p>