title: Problems with merging on-disk tables with millions of rows
body: <blockquote>
    <p>TypeError: Cannot serialize the column [date] because its data
    contents are [empty] object dtype.</p>
  </blockquote>
  
  <p>Hello SO! Currently have got two large HDFStore containing each one node, both the nodes doesn't fit in memory. The nodes don't contain NaN values. Now I would like to merge these two nodes using <a href="https://stackoverflow.com/questions/14614512/merging-two-tables-with-millions-of-rows-in-python/14617925#14617925">this</a>. First tested for a small store where all the data fits in one chunk and this was working OK. But now for the case where it has to merge chunk by chunk and it's giving me the following error: <code>TypeError: Cannot serialize the column [date], because its data contents are [empty] object dtype</code>.</p>
  
  <p>This is the code that I'm running.</p>
  
  <pre><code>&gt;&gt;&gt; import pandas as pd
  &gt;&gt;&gt; from pandas import HDFStore
  &gt;&gt;&gt; print pd.__version__
  0.12.0rc1
  
  &gt;&gt;&gt; h5_1 ='I:/Data/output/test8\\var1.h5'
  &gt;&gt;&gt; h5_3 ='I:/Data/output/test8\\var3.h5'
  &gt;&gt;&gt; h5_1temp = h5_1.replace('.h5','temp.h5')
  
  &gt;&gt;&gt; A = HDFStore(h5_1)
  &gt;&gt;&gt; B = HDFStore(h5_3)
  &gt;&gt;&gt; Atemp = HDFStore(h5_1temp)
  
  &gt;&gt;&gt; print A
  &lt;class 'pandas.io.pytables.HDFStore'&gt;
  File path: I:/Data/output/test8\var1.h5
  /var1            frame_table  (shape-&gt;12626172)
  &gt;&gt;&gt; print B
  &lt;class 'pandas.io.pytables.HDFStore'&gt;
  File path: I:/Data/output/test8\var3.h5
  /var3            frame_table  (shape-&gt;6313086)
  
  &gt;&gt;&gt; nrows_a = A.get_storer('var1').nrows
  &gt;&gt;&gt; nrows_b = B.get_storer('var3').nrows
  &gt;&gt;&gt; a_chunk_size = 500000
  &gt;&gt;&gt; b_chunk_size = 500000
  &gt;&gt;&gt; for a in xrange(int(nrows_a / a_chunk_size) + 1):
  ...     a_start_i = a * a_chunk_size
  ...     a_stop_i  = min((a + 1) * a_chunk_size, nrows_a)
  ...     a = A.select('var1', start = a_start_i, stop = a_stop_i)
  ...     for b in xrange(int(nrows_b / b_chunk_size) + 1):
  ...         b_start_i = b * b_chunk_size
  ...         b_stop_i = min((b + 1) * b_chunk_size, nrows_b)
  ...         b = B.select('var3', start = b_start_i, stop = b_stop_i)
  ...         Atemp.append('mergev13', pd.merge(a, b , left_index=True, right_index=True,how='inner'))
  
  ... 
  Traceback (most recent call last):
    File "&lt;interactive input&gt;", line 9, in &lt;module&gt;
    File "D:\Python27\lib\site-packages\pandas\io\pytables.py", line 658, in append
      self._write_to_group(key, value, table=True, append=True, **kwargs)
    File "D:\Python27\lib\site-packages\pandas\io\pytables.py", line 923, in _write_to_group
      s.write(obj = value, append=append, complib=complib, **kwargs)
    File "D:\Python27\lib\site-packages\pandas\io\pytables.py", line 3251, in write
      return super(AppendableMultiFrameTable, self).write(obj=obj.reset_index(), data_columns=data_columns, **kwargs)
    File "D:\Python27\lib\site-packages\pandas\io\pytables.py", line 2983, in write
      **kwargs)
    File "D:\Python27\lib\site-packages\pandas\io\pytables.py", line 2715, in create_axes
      raise e
  TypeError: Cannot serialize the column [date] because
  its data contents are [empty] object dtype
  </code></pre>
  
  <p>Things that I noticed, it mentions that I'm on pandas_version:= '0.10.1', however my pandas version is 0.12.0rc1. Further more some more specific information of the nodes:</p>
  
  <pre><code>&gt;&gt;&gt; A.select_column('var1','date').unique()
  array([2006001, 2006009, 2006017, 2006025, 2006033, 2006041, 2006049,
         2006057, 2006065, 2006073, 2006081, 2006089, 2006097, 2006105,
         2006113, 2006121, 2006129, 2006137, 2006145, 2006153, 2006161,
         2006169, 2006177, 2006185, 2006193, 2006201, 2006209, 2006217,
         2006225, 2006233, 2006241, 2006249, 2006257, 2006265, 2006273,
         2006281, 2006289, 2006297, 2006305, 2006313, 2006321, 2006329,
         2006337, 2006345, 2006353, 2006361], dtype=int64)
  
  &gt;&gt;&gt; B.select_column('var3','date').unique()
  array([2006001, 2006017, 2006033, 2006049, 2006065, 2006081, 2006097,
         2006113, 2006129, 2006145, 2006161, 2006177, 2006193, 2006209,
         2006225, 2006241, 2006257, 2006273, 2006289, 2006305, 2006321,
         2006337, 2006353], dtype=int64)
  
  &gt;&gt;&gt; A.get_storer('var1').levels
  ['x', 'y', 'date']
  
  &gt;&gt;&gt; A.get_storer('var1').attrs
  /var1._v_attrs (AttributeSet), 12 attributes:
     [CLASS := 'GROUP',
      TITLE := '',
      VERSION := '1.0',
      data_columns := ['date', 'y', 'x'],
      index_cols := [(0, 'index')],
      levels := ['x', 'y', 'date'],
      nan_rep := 'nan',
      non_index_axes := [(1, ['x', 'y', 'date', 'var1'])],
      pandas_type := 'frame_table',
      pandas_version := '0.10.1',
      table_type := 'appendable_multiframe',
      values_cols := ['values_block_0', 'date', 'y', 'x']]
  
  &gt;&gt;&gt; A.get_storer('var1').table
  /var1/table (Table(12626172,)) ''
    description := {
    "index": Int64Col(shape=(), dflt=0, pos=0),
    "values_block_0": Float64Col(shape=(1,), dflt=0.0, pos=1),
    "date": Int64Col(shape=(), dflt=0, pos=2),
    "y": Int64Col(shape=(), dflt=0, pos=3),
    "x": Int64Col(shape=(), dflt=0, pos=4)}
    byteorder := 'little'
    chunkshape := (3276,)
    autoIndex := True
    colindexes := {
      "date": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
      "index": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
      "y": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
      "x": Index(6, medium, shuffle, zlib(1)).is_CSI=False}
  
  &gt;&gt;&gt; B.get_storer('var3').levels
  ['x', 'y', 'date']
  
  &gt;&gt;&gt; B.get_storer('var3').attrs
  /var3._v_attrs (AttributeSet), 12 attributes:
     [CLASS := 'GROUP',
      TITLE := '',
      VERSION := '1.0',
      data_columns := ['date', 'y', 'x'],
      index_cols := [(0, 'index')],
      levels := ['x', 'y', 'date'],
      nan_rep := 'nan',
      non_index_axes := [(1, ['x', 'y', 'date', 'var3'])],
      pandas_type := 'frame_table',
      pandas_version := '0.10.1',
      table_type := 'appendable_multiframe',
      values_cols := ['values_block_0', 'date', 'y', 'x']]
  
  &gt;&gt;&gt; B.get_storer('var3').table
  /var3/table (Table(6313086,)) ''
    description := {
    "index": Int64Col(shape=(), dflt=0, pos=0),
    "values_block_0": Float64Col(shape=(1,), dflt=0.0, pos=1),
    "date": Int64Col(shape=(), dflt=0, pos=2),
    "y": Int64Col(shape=(), dflt=0, pos=3),
    "x": Int64Col(shape=(), dflt=0, pos=4)}
    byteorder := 'little'
    chunkshape := (3276,)
    autoIndex := True
    colindexes := {
      "date": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
      "index": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
      "y": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
      "x": Index(6, medium, shuffle, zlib(1)).is_CSI=False}
  
  &gt;&gt;&gt; print Atemp
  &lt;class 'pandas.io.pytables.HDFStore'&gt;
  File path: I:/Data/output/test8\var1temp.h5
  /mergev13            frame_table  (shape-&gt;823446)
  </code></pre>
  
  <p>Since chunksize is 500000 and shape of node in Atemp is 823446, gives me that at least one chunk is merged. But I cannot figure out where the error is coming from and I also run out of clues trying to discover where it's exactly going wrong. Any help is very much appreciated..</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>By reducing the chunksize of my test store it gives the same error. Of course not good, but now gives me possibility to share. Click <a href="https://files.myopera.com/m.vanhoek/files/data.zip" rel="nofollow noreferrer">here</a> for the code + HDFStores.</p>