title: Download, extract and read a gzip file in Python
body: <p>I'd like to download, extract and iterate over a text file in Python without having to create temporary files.</p>
  
  <p>basically, this pipe, but in python</p>
  
  <pre><code>curl ftp://ftp.theseed.org/genomes/SEED/SEED.fasta.gz | gunzip | processing step
  </code></pre>
  
  <p>Here's my code:</p>
  
  <pre><code>def main():
      import urllib
      import gzip
  
      # Download SEED database
      print 'Downloading SEED Database'
      handle = urllib.urlopen('ftp://ftp.theseed.org/genomes/SEED/SEED.fasta.gz')
  
  
      with open('SEED.fasta.gz', 'wb') as out:
          while True:
              data = handle.read(1024)
              if len(data) == 0: break
              out.write(data)
  
      # Extract SEED database
      handle = gzip.open('SEED.fasta.gz')
      with open('SEED.fasta', 'w') as out:
          for line in handle:
              out.write(line)
  
      # Filter SEED database
      pass
  </code></pre>
  
  <p>I don't want to use process.Popen() or anything because I want this script to be platform-independent.</p>
  
  <p>The problem is that the Gzip library only accepts filenames as arguments and not handles.  The reason for "piping" is that the download step only uses up ~5% CPU and it would be faster to run the extraction and processing at the same time.</p>
  
  <hr>
  
  <p><strong>EDIT</strong>:
  This won't work because</p>
  
  <blockquote>
    <p>"Because of the way gzip compression
    works, GzipFile needs to save its
    position and move forwards and
    backwards through the compressed file.
    This doesn't work when the “file” is a
    stream of bytes coming from a remote
    server; all you can do with it is
    retrieve bytes one at a time, not move
    back and forth through the data
    stream." - <a href="http://diveintopython.net/http_web_services/gzip_compression.html" rel="nofollow noreferrer">dive into python</a></p>
  </blockquote>
  
  <p>Which is why I get the error</p>
  
  <pre><code>AttributeError: addinfourl instance has no attribute 'tell'
  </code></pre>
  
  <p>So how does <code>curl url | gunzip | whatever</code> work?</p>