title: Python out of memory on large CSV file (numpy)
body: <p>I have a 3GB CSV file that I try to read with python, I need the median column wise.</p>
  
  <pre><code>from numpy import * 
  def data():
      return genfromtxt('All.csv',delimiter=',')
  
  data = data() # This is where it fails already.
  
  med = zeros(len(data[0]))
  data = data.T
  for i in xrange(len(data)):
      m = median(data[i])
      med[i] = 1.0/float(m)
  print med
  </code></pre>
  
  <p>The error that I get is this:</p>
  
  <pre><code>Python(1545) malloc: *** mmap(size=16777216) failed (error code=12)
  
  *** error: can't allocate region
  
  *** set a breakpoint in malloc_error_break to debug
  
  Traceback (most recent call last):
  
    File "Normalize.py", line 40, in &lt;module&gt;
  
    data = data()
  
    File "Normalize.py", line 39, in data
  
    return genfromtxt('All.csv',delimiter=',')
  
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
  packages/numpy/lib/npyio.py", line 1495, in genfromtxt
  
  for (i, line) in enumerate(itertools.chain([first_line, ], fhd)):
  
  MemoryError
  </code></pre>
  
  <p>I think it's just an out of memory error. I am running a 64bit MacOSX with 4GB of ram and both numpy and Python compiled in 64bit mode. </p>
  
  <p>How do I fix this? Should I try a distributed approach, just for the memory management? </p>
  
  <p>Thanks</p>
  
  <p>EDIT: Also tried with this but no luck...</p>
  
  <pre><code>genfromtxt('All.csv',delimiter=',', dtype=float16)
  </code></pre>