title: python subprocess continuously feed stdin and read stdout which is possibly empty
body: <p>I am trying to write a python script that optimises the parameters of a compiled c script that I wrote. The program (<code>dups</code>) is taking an input (using <code>fgets</code>) and prints the input if it is not in the set of previously seen inputs.</p>
  
  <p>My python script should continuously feed input to <code>dups</code> followed by an evaluation of it's output (which can be either the input or nothing).</p>
  
  <p>I am using subprocess writing to its <code>stdin</code> and reading from its <code>stdout</code>. However, using various solutions, the script hangs after reading the first line.</p>
  
  <p>The C program only writes to stdout and I know that it is flushing. Furthermore I am able to reproduce my error using a python script simulating <code>dups</code>.</p>
  
  <p>Sample code:</p>
  
  <p>Sample <code>dups</code> for easy reproduction:</p>
  
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/python
  import sys
  names = set()
  while True:
      try:
          x = raw_input()
      except EOFError:
          print "Unexpected EOF, quitting"
          break
      if x not in names:
          print x
          sys.stdout.flush()
      names.add(x)
  </code></pre>
  
  <p>Main script:</p>
  
  <pre class="lang-py prettyprint-override"><code>import subprocess
  import pty, os #for solution 4
  
  inputs = ['Alice', 'Alice', 'Bob', 'Jane', 'Bob', 'Jane', 'Alice', 'Mike']
  
  solution = 4
  
  arguments = ['./sample_script.py']
  
  popen_nargs = dict(stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE,
                     universal_newlines=True,
                     )
  
  if solution == 3:
      out_fname = "output.txt"
      f_w = open(out_fname,"wt")
      f_r = open(out_fname, "rt")
      popen_nargs['stdout'] = f_w
      popen_nargs['bufsize']=0
  
  if solution == 4:
      master, slave = pty.openpty()
      popen_nargs['stdout'] = slave
      stdout_handle = os.fdopen(master)
  
  child = subprocess.Popen(arguments, **popen_nargs)
  
  for inpt in inputs:
  
      print "Testing '" + inpt + "'"
      child.stdin.write(inpt +" \n")
      child.stdin.flush()
  
  
      print "Result:"
  
      #http://stackoverflow.com/questions/4417546/constantly-print-subprocess-output-while-process-is-running
      if solution == 0:
          for stdout_line in iter(child.stdout.readline, ""):
              print "-&gt; '", stdout_line, "'"
  
      #http://blog.endpoint.com/2015/01/getting-realtime-output-using-python.html
      elif solution == 1:
          while True:
              output = child.stdout.readline()
              if output == '' and child.poll() is not None:
                  break
              if output:
                  print "-&gt; '", output, "'"
  
      #https://gist.github.com/zhenyi2697/7229421
      elif solution == 2:
          output=''
          while True:
              result = child.poll()
              delta = child.stdout.read(1)
  
              if result is not None:
                  print 'terminated'
                  break
  
              if delta != ' ':
                  output = output + delta
              else:
                  if '%' in output:
                      print 'percentage is:'
                      print output
  
                  elif '/s' in output:
                      print 'speed is:'
                      print output
  
                  print "-&gt; '", output, "'"
                  output = ''
  
      #http://stackoverflow.com/questions/5419888/reading-from-a-frequently-updated-file
      elif solution == 3:
          f_w.flush()
          print "-&gt; '", f_r.read(), "'"
  
      print "end iteration"
  
      #http://stackoverflow.com/questions/13603334/repeatedly-write-to-stdin-and-read-stdout-of-a-subprocess-without-closing-it
      if solution == 4:
          print "-&gt; '", stdout_handle.readline(), "'"
  
  if solution == 3:
      f_w.close()
      f_r.close()
  
  # Close the program
  child.communicate()
  </code></pre>
  
  <p>Output:</p>
  
  <ul>
  <li><p>Solutions 0, 1 (don't terminate):</p>
  
  <pre><code>Testing 'Alice'
  Result:
  -&gt; ' Alice 
  '
  </code></pre></li>
  <li><p>Solution 2 (doest terminate):</p>
  
  <pre><code>Testing 'Alice'
  Result:
  -&gt; ' Alice '
  </code></pre></li>
  <li><p>Solution 3 (output.txt seems to be updated only in the end, despite bufsize=0):</p>
  
  <pre><code>Testing 'Alice'
  Result:
  -&gt; '  '
  end iteration
  Testing 'Alice'
  Result:
  -&gt; '  '
  end iteration
  
  ...
  
  Testing 'Mike'
  Result:
  -&gt; '  '
  end iteration
  </code></pre></li>
  <li><p>Solution 4 (seems promising, but <code>read</code> and <code>readline</code> do not work when there is no output hence it does not terminate): </p>
  
  <pre><code>Testing 'Alice'
  Result:
  -&gt; ' Alice 
  '
  end iteration
  Testing 'Alice'
  Result:
  -&gt; '
  </code></pre></li>
  </ul>