title: Evaluating HDF5: What limitations/features does HDF5 provide for modelling data?
body: <p>We are in evaluating technologies that we'll use to store data that we gather during the analysis of C/C++ code.  In the case of C++, the amount of data can be relatively large, ~20Mb per TU.</p>
  
  <p>After reading the following SO <a href="https://stackoverflow.com/questions/416436/what-to-put-in-a-binary-data-files-header/416937#416937">answer</a> it made me consider that <a href="http://www.hdfgroup.org/HDF5/" rel="nofollow noreferrer">HDF5</a> might be a suitable technology for us to use.  I was wondering if people here could help me answer a few initial questions that I have:</p>
  
  <ol>
  <li><p>Performance.  The general usage for the data will be write once and read "several" times, similar to the lifetime of a '.o' file generated by a compiler.  How does HDF5 compare against using something like an SQLite DB?  Is that even a reasonable comparison to make?</p></li>
  <li><p>Over time we will add to the information that we are storing, but will not necessarily want to re-distribute a completely new set of "readers" to support a new format.  After reading the user guide I understand that HDF5 is similar to XML or a DB, in that information is associated with a tag/column and so a tool built to read an older structure will just ignore the fields that it is not concerned with?  Is my understanding on this correct?</p></li>
  <li><p>A significant chunk of the information that we wish to write out will be a tree type of structure: scope hierarchy, type hierarchy etc.  Ideally we would model scopes as having parents, children etc. Is it possible to have one HDF5 object "point" to another?  If not, is there a standard technique to solve this problem using HDF5?  Or, as is required in a DB, do we need a unique key that would "link" one object to another with appropriate lookups when searching for the data?</p></li>
  </ol>
  
  <p>Many thanks!</p>