title: Python decompressing gzip chunk-by-chunk
body: <p>I've a memory- and disk-limited environment where I need to decompress the contents of a gzip file sent to me in string-based chunks (over xmlrpc binary transfer). However, using the zlib.decompress() or zlib.decompressobj()/decompress() both barf over the gzip header. I've tried offsetting past the gzip header (documented <a href="http://www.gzip.org/zlib/rfc-gzip.html">here</a>), but still haven't managed to avoid the barf. The gzip library itself only seems to support decompressing from files.</p>
  
  <p>The following snippet gives a simplified illustration of what I would like to do (except in real life the buffer will be filled from xmlrpc, rather than reading from a local file):</p>
  
  <pre><code>#! /usr/bin/env python
  
  import zlib
  
  CHUNKSIZE=1000
  
  d = zlib.decompressobj()
  
  f=open('23046-8.txt.gz','rb')
  buffer=f.read(CHUNKSIZE)
  
  while buffer:
    outstr = d.decompress(buffer)
    print(outstr)
    buffer=f.read(CHUNKSIZE)
  
  outstr = d.flush()
  print(outstr)
  
  f.close()
  </code></pre>
  
  <p>Unfortunately, as I said, this barfs with:</p>
  
  <pre><code>Traceback (most recent call last):
    File "./test.py", line 13, in &lt;module&gt;
      outstr = d.decompress(buffer)
  zlib.error: Error -3 while decompressing: incorrect header check 
  </code></pre>
  
  <p>Theoretically, I could feed my xmlrpc-sourced data into a StringIO and then use that as a fileobj for gzip.GzipFile(), however, in real life, I don't have memory available to hold the entire file contents in memory as well as the decompressed data. I really do need to process it chunk-by-chunk.</p>
  
  <p>The fall-back would be to change the compression of my xmlrpc-sourced data from gzip to plain zlib, but since that impacts other sub-systems I'd prefer to avoid it if possible.</p>
  
  <p>Any ideas?</p>