title: blocks - send input to python subprocess pipeline
body: <p>I'm testing subprocesses pipelines with python. I'm aware that I can do what the programs below do in python directly, but that's not the point. I just want to test the pipeline so I know how to use it.</p>
  
  <p>My system is Linux Ubuntu 9.04 with default python 2.6.</p>
  
  <p>I started with this <a href="http://docs.python.org/library/subprocess.html#replacing-shell-pipeline" rel="noreferrer">documentation example</a>.</p>
  
  <pre><code>from subprocess import Popen, PIPE
  p1 = Popen(["grep", "-v", "not"], stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  output = p2.communicate()[0]
  print output
  </code></pre>
  
  <p>That works, but since <code>p1</code>'s <code>stdin</code> is not being redirected, I have to type stuff in the terminal to feed the pipe. When I type <code>^D</code> closing stdin, I get the output I want.</p>
  
  <p>However, I want to send data to the pipe using a python string variable. First I tried writing on stdin:</p>
  
  <pre><code>p1 = Popen(["grep", "-v", "not"], stdin=PIPE, stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  p1.stdin.write('test\n')
  output = p2.communicate()[0] # blocks forever here
  </code></pre>
  
  <p>Didn't work. I tried using <code>p2.stdout.read()</code> instead on last line, but it also blocks. I added <code>p1.stdin.flush()</code> and <code>p1.stdin.close()</code> but it didn't work either. I Then I moved to communicate:</p>
  
  <pre><code>p1 = Popen(["grep", "-v", "not"], stdin=PIPE, stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  p1.communicate('test\n') # blocks forever here
  output = p2.communicate()[0] 
  </code></pre>
  
  <p>So that's still not it.</p>
  
  <p>I noticed that running a single process (like <code>p1</code> above, removing <code>p2</code>) works perfectly. And passing a file handle to <code>p1</code> (<code>stdin=open(...)</code>) also works. So the problem is:</p>
  
  <p>Is it possible to pass data to a pipeline of 2 or more subprocesses in python, without blocking? Why not?</p>
  
  <p>I'm aware I could run a shell and run the pipeline in the shell, but that's not what I want.</p>
  
  <hr>
  
  <p><strong>UPDATE 1</strong>: Following Aaron Digulla's hint below I'm now trying to use threads to make it work.</p>
  
  <p>First I've tried running p1.communicate on a thread.</p>
  
  <pre><code>p1 = Popen(["grep", "-v", "not"], stdin=PIPE, stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  t = threading.Thread(target=p1.communicate, args=('some data\n',))
  t.start()
  output = p2.communicate()[0] # blocks forever here
  </code></pre>
  
  <p>Okay, didn't work. Tried other combinations like changing it to <code>.write()</code> and also <code>p2.read()</code>. Nothing. Now let's try the opposite approach:</p>
  
  <pre><code>def get_output(subp):
      output = subp.communicate()[0] # blocks on thread
      print 'GOT:', output
  
  p1 = Popen(["grep", "-v", "not"], stdin=PIPE, stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  t = threading.Thread(target=get_output, args=(p2,)) 
  t.start()
  p1.communicate('data\n') # blocks here.
  t.join()
  </code></pre>
  
  <p>code ends up blocking somewhere. Either in the spawned thread, or in the main thread, or both. So it didn't work. If you know how to make it work it would make easier if you can provide working code. I'm trying here.</p>
  
  <hr>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p>Paul Du Bois answered below with some information, so I did more tests. 
  I've read entire <code>subprocess.py</code> module and got how it works. So I tried applying exactly that to code.</p>
  
  <p>I'm on linux, but since I was testing with threads, my first approach was to replicate the exact windows threading code seen on <code>subprocess.py</code>'s <code>communicate()</code> method, but for two processes instead of one. Here's the entire listing of what I tried:</p>
  
  <pre><code>import os
  from subprocess import Popen, PIPE
  import threading
  
  def get_output(fobj, buffer):
      while True:
          chunk = fobj.read() # BLOCKS HERE
          if not chunk:
              break
          buffer.append(chunk)
  
  p1 = Popen(["grep", "-v", "not"], stdin=PIPE, stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  
  b = [] # create a buffer
  t = threading.Thread(target=get_output, args=(p2.stdout, b))
  t.start() # start reading thread
  
  for x in xrange(100000):
      p1.stdin.write('hello world\n') # write data
      p1.stdin.flush()
  p1.stdin.close() # close input...
  t.join()
  </code></pre>
  
  <p>Well. It didn't work. Even after <code>p1.stdin.close()</code> was called, <code>p2.stdout.read()</code> still blocks.</p>
  
  <p>Then I tried the posix code on <code>subprocess.py</code>:</p>
  
  <pre><code>import os
  from subprocess import Popen, PIPE
  import select
  
  p1 = Popen(["grep", "-v", "not"], stdin=PIPE, stdout=PIPE)
  p2 = Popen(["cut", "-c", "1-10"], stdin=p1.stdout, stdout=PIPE)
  
  numwrites = 100000
  to_read = [p2.stdout]
  to_write = [p1.stdin]
  b = [] # create buffer
  
  while to_read or to_write:
      read_now, write_now, xlist = select.select(to_read, to_write, [])
      if read_now:
          data = os.read(p2.stdout.fileno(), 1024)
          if not data:
              p2.stdout.close()
              to_read = []
          else:
              b.append(data)
  
      if write_now:
          if numwrites &gt; 0:
              numwrites -= 1
              p1.stdin.write('hello world!\n'); p1.stdin.flush()
          else:
              p1.stdin.close()
              to_write = []
  
  print b
  </code></pre>
  
  <p>Also blocks on <code>select.select()</code>. By spreading <code>print</code>s around, I found out this:</p>
  
  <ul>
  <li>Reading is working. Code reads many times during execution.</li>
  <li>Writing is also working. Data is written to <code>p1.stdin</code>.</li>
  <li>At the end of <code>numwrites</code>, <code>p1.stdin.close()</code> is called.</li>
  <li>When <code>select()</code> starts blocking, only <code>to_read</code> has something, <code>p2.stdout</code>. <code>to_write</code> is already empty.</li>
  <li><code>os.read()</code> call always returns something, so <code>p2.stdout.close()</code> is never called.</li>
  </ul>
  
  <p><strong>Conclusion from both tests</strong>:  Closing the <code>stdin</code> of the first process on the pipeline (<code>grep</code> in the example) is not making it dump its buffered output to the next and die.</p>
  
  <p>No way to make it work?</p>
  
  <p>PS: I don't want to use a temporary file, I've already tested with files and I know it works. And I don't want to use windows.</p>