title: How do I use multiprocessing.Queue from a process with a pre-existing Pipe?
body: <p>I am trying to use <a href="https://docs.python.org/3/library/multiprocessing.html" rel="nofollow noreferrer"><code>multiprocessing</code></a> from inside another process that was spawned with <a href="https://docs.python.org/3/library/subprocess.html" rel="nofollow noreferrer"><code>Popen</code></a>. I want to be able to communicate between this process and a new child process, but this &quot;middle&quot; process has a polling read on the pipe with <em>its</em> parent, which seems to block execution of its child process.</p>
  <p>Here is my file structure:</p>
  <p><strong>entry.py</strong></p>
  <pre class="lang-py prettyprint-override"><code>import subprocess, threading, time, sys
  
  def start():
      # Create process 2
      worker = subprocess.Popen([sys.executable, &quot;-u&quot;, &quot;mproc.py&quot;],
              # When creating the subprocess with an open pipe to stdin and
              # subsequently polling that pipe, it blocks further communication
              # between subprocesses
              stdin=subprocess.PIPE,
              close_fds=False,)
      t = threading.Thread(args=(worker))
      t.start()
      time.sleep(4)
  
  if __name__ == '__main__':
      start()
  </code></pre>
  <p><strong>mproc.py</strong></p>
  <pre class="lang-py prettyprint-override"><code>import multiprocessing as mp
  import time, sys, threading
  
  def exit_on_stdin_close():
      try:
          while sys.stdin.read():
              pass
      except:
          pass
  
  
  def put_hello(q):
      # We never reach this line if exit_poll.start() is uncommented
      q.put(&quot;hello&quot;)
      time.sleep(2.4)
  
  
  def start():
      exit_poll = threading.Thread(target=exit_on_stdin_close, name=&quot;exit-poll&quot;)
      exit_poll.daemon = True
      # This daemon thread polling stdin blocks execution of subprocesses
      # But ONLY if running in another process with stdin connected
      # to its parent by PIPE
      exit_poll.start()
  
      ctx = mp.get_context('spawn')
  
      q = ctx.Queue()
      p = ctx.Process(target=put_hello, args=(q,))
  
      # Create process 3
      p.start()
      p.join()
      print(f&quot;result: {q.get()}&quot;)
  
  
  if __name__ == '__main__':
      start()
  </code></pre>
  <p>My desired behavior is that when running <code>entry.py</code>, <code>mproc.py</code> should run on a subprocess and be able to communicate with its own subprocess to get the <code>Queue</code> output, and this <em>does</em> happen if I don't start the <code>exit-poll</code> daemon thread:</p>
  <pre class="lang-sh prettyprint-override"><code>$ python -u entry.py
  result: hello
  </code></pre>
  <p>but if <code>exit-poll</code> <em>is</em> running, then process 3 blocks as soon as it's started. The <code>put_hello</code> method isn't even entered until the <code>exit-poll</code> thread ends.</p>
  <p>Is there a way to create a process 3 from process 2 and communicate between the two, even while the pipe between processes 1 and 2 is being used?</p>
  <p>Edit: I can only consistently reproduce this problem on Windows. On Linux (Ubuntu 20.04 WSL) the <code>Queues</code> are able to communicate even with <code>exit-poll</code> running, but <em>only</em> if I'm using the <code>spawn</code> multiprocessing context. If I change it to <code>fork</code> then I get the same behavior that I see on Windows.</p>