title: Is there any way to make fileinput.input() with openhook=hook_compressed performant?
body: <p>It seems that <code>fileinput.input</code> is at least twice as slow as <code>zcat</code>, even with buffering set. <strong>Question: Is there anything I can do to make it performant, without writing a bunch of code?</strong> What I did to test it was to take data from urandom,</p>
  
  <pre><code>"""generate.py"""
  import base64
  with open('/dev/urandom', 'rb') as f:
      for _ in xrange(102400):
          print(base64.b64encode(f.read(1024)))
  </code></pre>
  
  <p>run this and pipe the output through gzip,</p>
  
  <pre><code>&gt; python generate.py | gzip - &gt; test_input.gz
  </code></pre>
  
  <h2>zcat time</h2>
  
  <pre><code>&gt; time zcat test_input.gz &gt; /dev/null
  zcat test_input.gz &gt; /dev/null  1.56s user 0.02s system 99% cpu 1.576 total
  </code></pre>
  
  <h2>fileinput time</h2>
  
  <pre><code>&gt; time python -c 'import fileinput; list(fileinput.input(files=["test_input.gz"], openhook=fileinput.hook_compressed))'
  python -c   3.13s user 0.16s system 99% cpu 3.293 total
  </code></pre>
  
  <p>This isn't just <code>fileinput.input()</code> being slow, since when it reads from stdin it's fine,</p>
  
  <pre><code>&gt; time zcat test_input.gz | python -c 'import fileinput; list(fileinput.input())'
  zcat test_input.gz  1.64s user 0.04s system 96% cpu 1.736 total
  python -c 'import fileinput; list(fileinput.input())'  0.39s user 0.17s system 31% cpu 1.800 total
  </code></pre>
  
  <p>I messed with <code>bufsize=</code>, but no luck.</p>
  
  <h2>writing a lot of code</h2>
  
  <p>I poked around google thinking <code>gzip</code> itself was slow, and found if I do some manual buffering it's fine,</p>
  
  <pre><code>"""read_buffered_manual.py"""
  import gzip
  
  def input_buffered_manual(filename, buf_size=32 * 1024):
      fd = gzip.open(filename)
      try:
          remaining = ''
          while True:
              input_ = fd.read(buf_size)
              if not input_:
                  if remaining:
                      yield remaining
                  return
  
              lines = input_.split('\n')
              lines[0] = remaining + lines[0]
              remaining = lines.pop()
              for line in lines:
                  yield line
      finally:
          fd.close()
  
  for line in input_buffered_manual("test_input.gz"):
      print line
  </code></pre>
  
  <p>this is fast, in fact even faster than zcat,</p>
  
  <pre><code>&gt; time python read_buffered_manual.py &gt; /dev/null
  python read_buffered_manual.py &gt; /dev/null  1.40s user 0.04s system 99% cpu 1.461 total
  </code></pre>