title: Running an external program from python: piping without waiting for all output
body: <p>I have looked at <a href="https://stackoverflow.com/questions/89228/calling-an-external-command-in-python">Calling an external command in Python</a> and tried every possible way with subprocess and os.popen but nothing seems to work.</p>
  
  <p>If I try</p>
  
  <pre><code>import os
  stream = os.popen("program.ex -f file.dat | grep fish | head -4")
  </code></pre>
  
  <p>I get lines and lines of</p>
  
  <pre><code>grep: broken pipe
  </code></pre>
  
  <p>If I switch the <code>grep</code> and <code>head</code> commands around, it never gets to the grep command because the output from program.ex is prohibitively long (which is why I run with <code>head -4</code>).</p>
  
  <p>Of course the following fails because of the pipes:</p>
  
  <pre><code>import subprocess as sp
  cmd = "program.ex -f file.dat | grep fish | head -4"
  proc = sp.Popen(cmd.split(),stdout=sp.PIPE,stderr=sp.PIPE)
  stdout, stderr = proc.communicate()
  </code></pre>
  
  <p>So I tried breaking it down</p>
  
  <pre><code>cmd1 = "program.ex -f file.ex"
  cmd2 = "head -4"
  cmd3 = "grep fish"
  proc1 = sp.Popen(cmd1.split(),stdout=sp.PIPE,stderr=sp.PIPE)
  proc2 = sp.Popen(cmd2.split(),stdout=sp.PIPE,stdin=proc1.stdout)
  proc3 = sp.Popen(cmd3.split(),stdout=sp.PIPE,stdin=proc2.stdout)
  stdout, stderr = proc1.communicate()
  </code></pre>
  
  <p>which does run, except it gets stuck on cmd1 because the output from program.ex is prohibitively long.</p>
  
  <p>Finally I tried hiding it in an external shell script and fortran program, but the fortran program does a</p>
  
  <pre><code>call system("program.ex -f file.dat | grep fish | head -4")
  </code></pre>
  
  <p>and I guess this messes up python again.</p>
  
  <p>Note: If I do this directly in the terminal, it doesn't need to get the whole output from program.ex and the command finishes instantly.</p>
  
  <p>So, my question is:</p>
  
  <p><strong>How can I get the above command to run in python like it does in the terminal (ie, head and grep the output from program.ex without needing to wait for all the output from program.ex)?</strong></p>
  
  <p>Help is greatly appreciated!</p>
  
  <p>Edit:</p>
  
  <p>I also tried with <code>shell=True</code>:</p>
  
  <pre><code>import subprocess as sp
  cmd = "program.ex -f file.dat | head -4 | grep fish"
  proc = sp.Popen(cmd.split(),stdout=sp.PIPE,stderr=sp.PIPE,shell=True)
  stdout, stderr = proc.communicate()
  </code></pre>
  
  <p>which does run, and while stderr has expected (un-needed) content, stdout is empty. If I replace the above <code>cmd</code> variable with the name of a fortran program which calls the system command instead, then it hangs on program.ex again, probably waiting for all the output to finish.</p>