title: Why doesn't Python's mmap work with large files?
body: <p><strong>[Edit: This problem applies only to 32-bit systems.  If your computer, your OS and your python implementation are 64-bit, then mmap-ing huge files works reliably and is extremely efficient.]</strong></p>
  
  <p>I am writing a module that amongst other things allows bitwise read access to files. The files can potentially be large (hundreds of GB) so I wrote a simple class that lets me treat the file like a string and hides all the seeking and reading.</p>
  
  <p>At the time I wrote my wrapper class I didn't know about the <a href="http://docs.python.org/library/mmap.html" rel="noreferrer">mmap module</a>. On reading the documentation for mmap I thought <em>"great - this is just what I needed, I'll take out my code and replace it with an mmap. It's probably much more efficient and it's always good to delete code."</em></p>
  
  <p>The problem is that mmap doesn't work for large files! This is very surprising to me as I thought it was perhaps the most obvious application. If the file is above a few gigabytes then I get an <code>EnvironmentError: [Errno 12] Cannot allocate memory</code>. This only happens with a 32-bit Python build so it seems it is running out of address space, but I can't find any documentation on this.</p>
  
  <p>My code is just</p>
  
  <pre><code>f = open('somelargefile', 'rb')
  map = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
  </code></pre>
  
  <p>So my question is <strong>am I missing something obvious here?</strong> Is there a way to get mmap to work portably on large files or should I go back to my naïve file wrapper?</p>
  
  <hr>
  
  <p>Update: There seems to be a feeling that the Python mmap should have the same restrictions as the POSIX mmap. To better express my frustration here is a simple class that has a small part of the functionality of mmap.</p>
  
  <pre><code>import os
  
  class Mmap(object):
      def __init__(self, f):
          """Initialise with a file object."""
          self.source = f
  
      def __getitem__(self, key):
          try:
              # A slice
              self.source.seek(key.start, os.SEEK_SET)
              return self.source.read(key.stop - key.start)
          except AttributeError:
              # single element
              self.source.seek(key, os.SEEK_SET)
              return self.source.read(1)
  </code></pre>
  
  <p>It's read-only and doesn't do anything fancy, but I can do this just the same as with an mmap:</p>
  
  <pre><code>map2 = Mmap(f)
  print map2[0:10]
  print map2[10000000000:10000000010]
  </code></pre>
  
  <p>except that there are no restrictions on filesize. Not too difficult really...</p>