title: In Linux, How to Compress Input Stream to Multiple Output Files?
body: <p>I know how to do this:</p>
  
  <pre><code>commandGeneratingLotsOfSTDOUT | bzip2 -z -c &gt; compressed.bz2
  </code></pre>
  
  <p>I also know how to do this:</p>
  
  <pre><code>commandGeneratingLotsOfSTDOUT | split -l 1000000
  </code></pre>
  
  <p>But I don't know how to do this:</p>
  
  <pre><code>commandGeneratingLotsOfSTDOUT | split -l 1000000 -compressCommand "bzip2 -z -c"
  </code></pre>
  
  <p>In case the above isn't already 100% clear, I am running a command that generates a terabyte or two of output. I want the output to be split into chunks of N lines (1 million in this case), and each chunk to be bzip2 compressed and stored in a file.</p>
  
  <p>Right now what I do is this:</p>
  
  <pre><code>commandGeneratingLotsOfSTDOUT | split -l 1000000
  foreach fileGenerated { bzip2 -z thatFile }
  </code></pre>
  
  <p>This adds an extra write to disk and read from disk (and write to disk again, albeit compressed) for every single file! Since the files are all bigger than RAM, this translates to actual disk usage.</p>