title: Random access to gzipped files?
body: <p>I have a very large file compressed with gzip sitting on disk. The production environment is &quot;Cloud&quot;-based, so the storage performance is terrible, but CPU is fine. Previously, our data processing pipeline began with <code>gzip -dc</code> streaming the data off the disk.</p>
  <p>Now, in order to parallelise the work, I want to run multiple pipelines that each take a pair of byte offsets - start and end - and take that chunk of the file. With a plain file this could be achieved with <code>head</code> and <code>tail</code>, but I'm not sure how to do it efficiently with a compressed file; if I <code>gzip -dc</code> and pipe into <code>head</code>, the offset pairs that are toward the end of the file will involve wastefully seeking through the whole file as it's slowly decompressed.</p>
  <p>So my question is really about the gzip algorithm - is it theoretically possible to seek to a byte offset in the underlying file or get an arbitrary chunk of it, without the full implications of decompressing the entire file up to that point? If not, how else might I efficiently partition a file for &quot;random&quot; access by multiple processes while minimising the I/O throughput overhead?</p>