title: Equivalent of set -o pipefail in Python?
body: <p>I have a handful of Python scripts each of which make heavy use of sorting, uniq-ing, counting, gzipping and gunzipping, and awking.  As a first run through the code I've used <code>subprocess.call</code> with (yes I know of the security risks that's why I said it is a first pass) <code>shell=True</code>.  I have a little helper function:</p>
  
  <pre><code>def do(command):
      start = datetime.now()
      return_code = call(command, shell=True)
      print 'Completed in', str(datetime.now() - start), 'ms, return code =', return_code
      if return_code != 0:
          print 'Failure: aborting with return code %d' % return_code
          sys.exit(return_code)
  </code></pre>
  
  <p>Scripts make use of this helper as in the following snippets:</p>
  
  <pre><code>do('gunzip -c %s | %s | sort -u | %s &gt; %s' % (input, parse, flatten, output))
  do("gunzip -c %s | grep 'en$' | cut -f1,2,4 -d\|| %s &gt; %s" % (input, parse, output))
  do('cat %s | %s | gzip -c &gt; %s' % (input, dedupe, output))
  do("awk -F ' ' '{print $%d,$%d}' %s | sort -u | %s | gzip -c &gt; %s" % params)
  do('gunzip -c %s | %s | gzip -c &gt; %s' % (input, parse, output))
  do('gunzip -c %s | %s &gt; %s' % (input, parse, collection))
  do('%s &lt; %s &gt;&gt; %s' % (parse, supplement, collection))
  do('cat %s %s | sort -k 2 | %s | gzip -c &gt; %s' % (source,other_source,match,output)
  </code></pre>
  
  <p>And there are many more like these, some with even longer pipelines.</p>
  
  <p>One issue I notice is that when a command early in a pipeline fails, the whole command will still succeed with exit status 0.  In bash I fix this with</p>
  
  <pre><code>set -o pipefail
  </code></pre>
  
  <p>but I do not see how this can be done in Python.  I suppose I could put in an explicit call to bash but that seems wrong.  Is it?</p>
  
  <p>In lieu of an answer to that specific question, I'd love to hear alternatives to implementing this kind of code in pure Python without resorting to <code>shell=True</code>.  But when I attempt to use <code>Popen</code> and <code>stdout=PIPE</code> the code size blows up.  There is something nice about writing pipelines on one line as a string, but if anyone knows an elegant multiline "proper and secure" way to do this in Python I would love to hear it!</p>
  
  <p>An aside: none of these scripts ever take user input; they run batch jobs on a machine with a known shell which is why I actually ventured into the evil <code>shell=True</code> just to see how things would look.  And they do look pretty easy to read and the code seems <em>so</em> concise!  How does one remove the <code>shell=True</code> and run these long pipelines in raw Python while still getting the advantages of aborting the process if an early component fails?</p>