title: Memory error when using pandas read_csv
body: <p>I am trying to do something fairly simple, reading a large csv file into a pandas dataframe.</p>
  
  <pre><code>data = pandas.read_csv(filepath, header = 0, sep = DELIMITER,skiprows = 2)
  </code></pre>
  
  <p>The code either fails with a <code>MemoryError</code>, or just never finishes.</p>
  
  <p>Mem usage in the task manager stopped at 506 Mb and after 5 minutes of no change and no CPU activity in the process I stopped it.</p>
  
  <p>I am using pandas version 0.11.0.</p>
  
  <p>I am aware that there used to be a memory problem with the file parser, but according to <strong><a href="http://wesmckinney.com/blog/?p=543">http://wesmckinney.com/blog/?p=543</a></strong> this should have been fixed.</p>
  
  <p>The file I am trying to read is 366 Mb, the code above works if I cut the file down to something short (25 Mb).</p>
  
  <p>It has also happened that I get a pop up telling me that it can't write to address 0x1e0baf93... </p>
  
  <p>Stacktrace:</p>
  
  <pre><code>Traceback (most recent call last):
    File "F:\QA ALM\Python\new WIM data\new WIM data\new_WIM_data.py", line 25, in
   &lt;module&gt;
      wimdata = pandas.read_csv(filepath, header = 0, sep = DELIMITER,skiprows = 2
  )
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\io\parsers.py"
  , line 401, in parser_f
      return _read(filepath_or_buffer, kwds)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\io\parsers.py"
  , line 216, in _read
      return parser.read()
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\io\parsers.py"
  , line 643, in read
      df = DataFrame(col_dict, columns=columns, index=index)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\frame.py"
  , line 394, in __init__
      mgr = self._init_dict(data, index, columns, dtype=dtype)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\frame.py"
  , line 525, in _init_dict
      dtype=dtype)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\frame.py"
  , line 5338, in _arrays_to_mgr
      return create_block_manager_from_arrays(arrays, arr_names, axes)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\internals
  .py", line 1820, in create_block_manager_from_arrays
      blocks = form_blocks(arrays, names, axes)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\internals
  .py", line 1872, in form_blocks
      float_blocks = _multi_blockify(float_items, items)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\internals
  .py", line 1930, in _multi_blockify
      block_items, values = _stack_arrays(list(tup_block), ref_items, dtype)
    File "C:\Program Files\Python\Anaconda\lib\site-packages\pandas\core\internals
  .py", line 1962, in _stack_arrays
      stacked = np.empty(shape, dtype=dtype)
  MemoryError
  Press any key to continue . . .
  </code></pre>
  
  <hr>
  
  <p>A bit of background - I am trying to convince people that Python can do the same as R. For this I am trying to replicate an R script that does</p>
  
  <pre><code>data &lt;- read.table(paste(INPUTDIR,config[i,]$TOEXTRACT,sep=""), HASHEADER, DELIMITER,skip=2,fill=TRUE)
  </code></pre>
  
  <p>R not only manages to read the above file just fine, it even reads several of these files in a for loop (and then does some stuff with the data). If Python does have a problem with files of that size I might be fighting a loosing battle...</p>