title: MemoryError when trying to concat huge dataframes from read_csv
body: <p>I have a large data set which contains approximate 109G data which are separated into 245 .vsc.gz files. I tried to open each of them and then join all of them. The first step is OK I successfully achieved what I want but later when I try to concat them I received the following error.`</p>
  
  <pre><code>Traceback (most recent call last):
    File "CleanDataVR2.py", line 92, in &lt;module&gt;
      df_concated = pd.concat(dl)
    File "/share/apps/anaconda/2/2.5.0/lib/python2.7/site-packages/pandas/tools/merge.py", line 835, in concat
      return op.get_result()
    File "/share/apps/anaconda/2/2.5.0/lib/python2.7/site-packages/pandas/tools/merge.py", line 1025, in get_result
      concat_axis=self.axis, copy=self.copy)
    File "/share/apps/anaconda/2/2.5.0/lib/python2.7/site-packages/pandas/core/internals.py", line 4474, in concatenate_block_managers
      for placement, join_units in concat_plan]
    File "/share/apps/anaconda/2/2.5.0/lib/python2.7/site-packages/pandas/core/internals.py", line 4579, in concatenate_join_units
      concat_values = com._concat_compat(to_concat, axis=concat_axis)
    File "/share/apps/anaconda/2/2.5.0/lib/python2.7/site-packages/pandas/core/common.py", line 2741, in _concat_compat
      return np.concatenate(to_concat, axis=axis)
  MemoryError
  </code></pre>
  
  <p>And then is my code.</p>
  
  <pre><code>dl= []
  dtype_dict= {'#RIC': 'str', 'Date[G]':'str', 'Time[G]':'str', 'Price': 'float16', 'Volume': 'float16'}
  
  for f in file_list:
      df=pd.read_csv(f, compression='gzip',header=0, sep=',', 
                 quotechar='"',usecols= 
                 ['#RIC','Date[G]','Time[G]','Price','Volume'],
                 dtype=dtype_dict,low_memory=False)
      df= df.dropna(subset=['Price']) 
      df['Tage'] = df['Volume'].map(volume_filter)
      dl.append(df)
  df_concated = pd.concat(dl)
  </code></pre>
  
  <p>That function named <code>volume_filter</code>is just a function I wrote to get the column named <code>Tage</code>. Since this part works well I didn't put it here. Each data frames have the same structure as follow.</p>
  
  <pre><code>            #RIC      Date[G]       Time[G]    Price  Volume   Tage
  0         URKAq.L  01-SEP-2008  11:19:46.800   45.000   152.0   T200
  8         URKAq.L  01-SEP-2008  11:28:53.769   45.000  2848.0  T3000
  9         URKAq.L  01-SEP-2008  11:28:53.769   45.000  1725.0  T2000
  11        URKAq.L  01-SEP-2008  11:28:53.844   45.000   427.0   T500
  13        URKAq.L  01-SEP-2008  11:28:53.898   45.000   450.0   T500
  15        URKAq.L  01-SEP-2008  11:28:53.981   45.000   200.0   T200
  20        URKAq.L  01-SEP-2008  11:28:54.124   45.000   850.0   T900
  21        URKAq.L  01-SEP-2008  11:28:54.124   45.000  1073.0  T2000
  24        URKAq.L  01-SEP-2008  11:28:54.329   45.000   200.0   T200
  25        URKAq.L  01-SEP-2008  11:28:54.617   44.965   310.0   T400
  26        URKAq.L  01-SEP-2008  11:28:54.617   44.965   310.0   T400
  29        URKAq.L  01-SEP-2008  11:29:04.522   45.025   620.0   T700
  30        URKAq.L  01-SEP-2008  11:29:04.769   45.025   620.0   T700
  31        URKAq.L  01-SEP-2008  11:30:21.974   45.000  2800.0  T3000
  32        URKAq.L  01-SEP-2008  11:30:21.974   45.000   700.0   T700
  35        URKAq.L  01-SEP-2008  11:30:22.036   45.000   679.0   T700
  39        URKAq.L  01-SEP-2008  11:30:22.110   45.000   200.0   T200
  40        URKAq.L  01-SEP-2008  11:30:22.114   45.000   250.0   T300
  42        URKAq.L  01-SEP-2008  11:30:22.405   45.025   243.0   T300
  43        URKAq.L  01-SEP-2008  11:30:22.663   45.025   243.0   T300
  44        URKAq.L  01-SEP-2008  11:30:23.737   45.000  2550.0  T3000
  47        URKAq.L  01-SEP-2008  11:30:23.769   45.000  1500.0  T2000
  51        URKAq.L  01-SEP-2008  11:30:23.769   44.920   200.0   T200
  52        URKAq.L  01-SEP-2008  11:30:23.856   44.900   150.0   T200
  54        URKAq.L  01-SEP-2008  11:30:25.101   45.000  1901.0  T2000
  56        URKAq.L  01-SEP-2008  11:30:25.145   44.900   650.0   T700
  58        URKAq.L  01-SEP-2008  11:30:25.145   44.900   200.0   T200
  64        URKAq.L  01-SEP-2008  11:30:37.648   44.950   195.0   T200
  65        URKAq.L  01-SEP-2008  11:30:37.829   44.950   195.0   T200
  68        URKAq.L  01-SEP-2008  11:30:47.743   44.950  1031.0  T2000
  </code></pre>
  
  <p>I tried one solution of @B. M.</p>
  
  <pre><code>def byhand(dfs):
      mtot=0
      with open('df_all.bin','wb') as f:
          for df in dfs:
              m,n =df.shape
              mtot += m
              f.write(df.values.tobytes())
              typ=df.values.dtype                
      #del dfs
      with open('df_all.bin','rb') as f:
          buffer=f.read()
          data=np.frombuffer(buffer,dtype=typ).reshape(mtot,n)
          df_all=pd.DataFrame(data=data,columns=list(range(n))) 
      os.remove('df_all.bin')
      return df_all
  </code></pre>
  
  <p>But I still got the following error.</p>
  
  <pre><code>Traceback (most recent call last):
    File "CleanDataVR2.py", line 107, in &lt;module&gt;
      byhand(dl)
    File "CleanDataVR2.py", line 102, in byhand
      data=np.frombuffer(buffer,dtype=typ).reshape(mtot,n)
  ValueError: cannot create an OBJECT array from memory buffer
  </code></pre>
  
  <p>Right now I'm really confused. I have run my own code with 2 files in a supercomputer and everthing is fun. Why I there is an error with more files? Plus I have used 5*256GB memory which should be enough.</p>