title: Storing .csv in HDF5 pandas
body: <p>I was experimenting with HDF and it seems pretty great because my data is not normalized and it contains a lot of text. I love being able to query when I read data into pandas. </p>
  
  <pre><code>    loc2 = r'C:\\Users\Documents\\'
      (my dataframe with data is called 'export')
      hdf = HDFStore(loc2+'consolidated.h5')
      hdf.put('raw', export, format= 'table', complib= 'blosc', complevel=9, data_columns = True, append = True)
  </code></pre>
  
  <p>21 columns and about 12 million rows so far and I will add about 1 million rows per month.</p>
  
  <ul>
  <li>1 Date column [I convert this to datetime64]</li>
  <li>2 Datetime columns (one of them for each row and the other one is null about 70% of the time) [I convert this to datetime64]</li>
  <li>9 text columns [I convert these to categorical which saves a ton of room]</li>
  <li>1 float column</li>
  <li>8 integer columns, 3 of these can reach a max of maybe a couple of hundred and the other 5 can only be 1 or 0 values</li>
  </ul>
  
  <p>I made a nice small h5 table and it was perfect until I tried to append more data to it (literally just one day of data since I am receiving daily raw .csv files). I received errors which showed that the dtypes were not matching up for each column although I used the same exact ipython notebook.</p>
  
  <ul>
  <li>Is my hdf.put code correct? If I have append = True does that mean it will create the file if it does not exist, but append the data if it does exist? I will be appending to this file everyday basically. </li>
  <li>For columns which only contain 1 or 0, should I specify a dtype like int8 or int16 - will this save space or should I keep it at int64? It looks like some of my columns are randomly float64 (although no decimals) and int64. I guess I need to specify the dtype for each column individually. Any tips?</li>
  <li>I have no idea what blosc compression is. Is that the most efficient one to use? Any recommendations here? This file is mainly used to quickly read data into a dataframe to join to other .csv files which Tableau is connected to</li>
  </ul>