title: Pandas MemoryError while pd.concat
body: <p>I am using pandas to read csv's.</p>
  
  <pre><code>df_from_each_file = (pd.read_csv(StringIO(f)), compression='gzip', dtype=str) for f in all_files)
  final_df = pd.concat(df_from_each_file, ignore_index=True)
  </code></pre>
  
  <p>total rows in all_files is about 90,00,000 though each file is of smaller size.</p>
  
  <p>When pd.concat runs it is failing citing <code>Memory Error</code>.</p>
  
  <p>The system has 16 GB of RAM and 16 CPU's 2 GHZ each.Is the memory insufficent here?
  Is there anything else i can do to remove MemoryError?</p>
  
  <p>i read about chunksize etc but each file is small and that should not be a problem.How can <code>concat</code> be made memoryerror free?</p>
  
  <p>This is the traceback.</p>
  
  <pre><code>final_df = pd.concat(df_from_each_file, ignore_index=True)
  File "/home/jenkins/fsroot/workspace/ric-dev-sim-2/VENV/lib/python2.7/site-packages/pandas/tools/merge.py", line 1326, in concat
  return op.get_result()
  File "/home/jenkins/fsroot/workspace/ric-dev-sim-2/VENV/lib/python2.7/site-packages/pandas/tools/merge.py", line 1517, in get_result
  copy=self.copy)
  File "/home/jenkins/fsroot/workspace/ric-dev-sim-2/VENV/lib/python2.7/site-packages/pandas/core/internals.py", line 4797, in concatenate_block_managers
  placement=placement) for placement, join_units in concat_plan]
  File "/home/jenkins/fsroot/workspace/ric-dev-sim-2/VENV/lib/python2.7/site-packages/pandas/core/internals.py", line 4902, in concatenate_join_units
  concat_values = _concat._concat_compat(to_concat, axis=concat_axis)
  File "/home/jenkins/fsroot/workspace/ric-dev-sim-2/VENV/lib/python2.7/site-packages/pandas/types/concat.py", line 165, in _concat_compat
  return np.concatenate(to_concat, axis=axis)
  MemoryError
  </code></pre>
  
  <p>df.info for 1 file is</p>
  
  <pre><code>dtype: object&lt;class 'pandas.core.frame.DataFrame'&gt;
  RangeIndex: 12516 entries, 0 to 12515
  Columns: 322 entries, #RIC to Reuters Classification Scheme.1
  dtypes: object(322)
  memory usage: 30.7+ MB
  None
  </code></pre>