title: Python MemoryError: cannot allocate array memory
body: <p>I've got a 250 MB CSV file I need to read with ~7000 rows and ~9000 columns. Each row represents an image, and each column is a pixel (greyscale value 0-255)</p>
  
  <p>I started with a simple <code>np.loadtxt("data/training_nohead.csv",delimiter=",")</code> but this gave me a memory error. I thought this was strange since I'm running 64-bit Python with 8 gigs of memory installed and it died after using only around 512 MB.</p>
  
  <p>I've since tried SEVERAL other tactics, including:</p>
  
  <ol>
  <li><code>import fileinput</code> and read one line at a time, appending them to an array</li>
  <li><code>np.fromstring</code> after reading in the entire file</li>
  <li><code>np.genfromtext</code></li>
  <li>Manual parsing of the file (since all data is integers, this was fairly easy to code)</li>
  </ol>
  
  <p>Every method gave me the same result. MemoryError around 512 MB. Wondering if there was something special about 512MB, I created a simple test program which filled up memory until python crashed:</p>
  
  <pre><code>str = " " * 511000000 # Start at 511 MB
  while 1:
      str = str + " " * 1000 # Add 1 KB at a time
  </code></pre>
  
  <p>Doing this didn't crash until around 1 gig. I also, just for fun, tried: <code>str = " " * 2048000000</code> (fill 2 gigs) - this ran without a hitch. Filled the RAM and never complained. So the issue isn't the total amount of RAM I can allocate, but seems to be how many TIMES I can allocate memory...</p>
  
  <p>I google'd around fruitlessly until I found this post: <a href="https://stackoverflow.com/questions/8956832/python-out-of-memory-on-large-csv-file-numpy">Python out of memory on large CSV file (numpy)</a></p>
  
  <p>I copied the code from the answer exactly:</p>
  
  <pre><code>def iter_loadtxt(filename, delimiter=',', skiprows=0, dtype=float):
      def iter_func():
          with open(filename, 'r') as infile:
              for _ in range(skiprows):
                  next(infile)
              for line in infile:
                  line = line.rstrip().split(delimiter)
                  for item in line:
                      yield dtype(item)
          iter_loadtxt.rowlength = len(line)
  
      data = np.fromiter(iter_func(), dtype=dtype)
      data = data.reshape((-1, iter_loadtxt.rowlength))
      return data
  </code></pre>
  
  <p>Calling <code>iter_loadtxt("data/training_nohead.csv")</code> gave a slightly different error this time:</p>
  
  <pre><code>MemoryError: cannot allocate array memory
  </code></pre>
  
  <p>Googling this error I only found one, not so helpful, post: <a href="https://stackoverflow.com/questions/3717418/memory-error-memoryerror-when-creating-a-boolean-numpy-array-python">Memory error (MemoryError) when creating a boolean NumPy array (Python)</a></p>
  
  <p>As I'm running Python 2.7, this was not my issue. Any help would be appreciated.</p>