title: Why does my python process use up so much memory?
body: <p>I'm working on a project that involves using python to read, process and write files that are sometimes as large as a few hundred megabytes. The program fails occasionally when I try to process some particularly large files. It does not say 'memory error', but I suspect that is the problem (in fact it gives no reason at all for failing'). </p>
  
  <p>I've been testing the code on smaller files and watching 'top' to see what memory usage is like, and it typically reaches 60%. top says that I have 4050352k total memory, so 3.8Gb.</p>
  
  <p>Meanwhile I'm trying to track memory usage within python itself (see my question from <a href="https://stackoverflow.com/questions/11784329/python-memory-usage-of-numpy-arrays">yesterday</a>) with the following little bit of code:</p>
  
  <pre><code>mem = 0
  for variable in dir():
      variable_ = vars()[variable]
      try: 
          if str(type(variable_))[7:12] == 'numpy':
              numpy_ = True
          else:
              numpy_ = False
      except:
          numpy_ = False
      if numpy_:
          mem_ = variable_.nbytes
      else:
          mem_ = sys.getsizeof(variable)
      mem += mem_
      print variable+ type: '+str(type(variable_))+' size: '+str(mem_)
  print 'Total: '+str(mem)
  </code></pre>
  
  <p>Before I run that block I set all the variables I don't need to None, close all files and figures, etc etc. After that block I use subprocess.call() to run a fortran program that is required for the next stage of processing. Looking at top while the fortran program is running shows that the fortran program is using ~100% of the cpu, and ~5% of the memory, and that python is using 0% of cpu and 53% of memory. However my little snippet of code tells me that all of the variables in python add up to only 23Mb, which ought to be ~0.5%.</p>
  
  <p>So what's happening? I wouldn't expect that little snippet to give me a spot on memory usage, but it ought to be accurate to within a few Mb surely? Or is it just that top doesn't notice the memory has been relinquished, but that it is available to other programs that need it if necessary? </p>
  
  <p>As requested, here's a simplified part of the code that is using up all the memory (file_name.cub is an ISIS3 cube, it's a file that contains 5 layers (bands) of the same map, the first layer is spectral radiance, the next 4 have to do with latitude, longitude, and other details. It's an image from Mars that I'm trying to process. StartByte is a value I previously read from the .cub file's ascii header telling me the beginning byte of the data, Samples and Lines are the dimensions of the map, also read from the header.):</p>
  
  <pre><code>latitude_array = 'cheese'   # It'll make sense in a moment
  f_to = open('To_file.dat','w') 
  
  f_rad = open('file_name.cub', 'rb')
  f_rad.seek(0)
  header=struct.unpack('%dc' % (StartByte-1), f_rad.read(StartByte-1))
  header = None    
  #
  f_lat = open('file_name.cub', 'rb')
  f_lat.seek(0)
  header=struct.unpack('%dc' % (StartByte-1), f_lat.read(StartByte-1))
  header = None 
  pre=struct.unpack('%df' % (Samples*Lines), f_lat.read(Samples*Lines*4))
  pre = None
  #
  f_lon = open('file_name.cub', 'rb')
  f_lon.seek(0)
  header=struct.unpack('%dc' % (StartByte-1), f_lon.read(StartByte-1))
  header = None 
  pre=struct.unpack('%df' % (Samples*Lines*2), f_lon.read(Samples*Lines*2*4))
  pre = None
  # (And something similar for the other two bands)
  # So header and pre are just to get to the right part of the file, and are 
  # then set to None. I did try using seek(), but it didn't work for some
  # reason, and I ended up with this technique.
  for line in range(Lines):
      sample_rad = struct.unpack('%df' % (Samples), f_rad.read(Samples*4))
      sample_rad = np.array(sample_rad)
      sample_rad[sample_rad&lt;-3.40282265e+38] = np.nan  
      # And Similar lines for all bands
      # Then some arithmetic operations on some of the arrays
      i = 0
      for value in sample_rad:
          nextline = sample_lat[i]+', '+sample_lon[i]+', '+value # And other stuff
          f_to.write(nextline)
          i += 1
      if radiance_array == 'cheese':  # I'd love to know a better way to do this!
          radiance_array = sample_rad.reshape(len(sample_rad),1)
      else:
          radiance_array = np.append(radiance_array, sample_rad.reshape(len(sample_rad),1), axis=1)
          # And again, similar operations on all arrays. I end up with 5 output arrays
          # with dimensions ~830*4000. For the large files they can reach ~830x20000
  f_rad.close()
  f_lat.close()
  f_to.close()   # etc etc 
  sample_lat = None  # etc etc
  sample_rad = None  # etc etc
  
  #
  plt.figure()
  plt.imshow(radiance_array)
  # I plot all the arrays, for diagnostic reasons
  
  plt.show()
  plt.close()
  
  radiance_array = None  # etc etc
  # I set all arrays apart from one (which I need to identify the 
  # locations of nan in future) to None
  
  # LOCATION OF MEMORY USAGE MONITOR SNIPPET FROM ABOVE
  </code></pre>
  
  <p>So I lied in the comments about opening several files, it's many instances of the same file. I only continue with one array that isn't set to None, and it's size is ~830x4000, though this somehow constitutes 50% of my available memory. I've also tried gc.collect, but no change. I'd be very happy to hear any advice on how I could improve on any of that code (related to this problem or otherwise).</p>
  
  <p>Perhaps I should mention: originally I was opening the files in full (i.e. not line by line as above), doing it line by line was an initial attempt to save memory.</p>