title: Asynchronously wait for multiprocessing Queue in main process
body: <p>I have the following scenario: multiple worker processes send events about their current status to an event dispatcher. This event dispatcher then needs to process all the events if we are in the main process or signal the event dispatcher of the main process to handle these events if we are in a worker process.</p>
  
  <p>The main crux here is that event handling must also be in the main <em>thread</em> of the main process, so I can't just run a while True loop inside a thread and wait for messages from worker processes there.</p>
  
  <p>So what I have is this:</p>
  
  <pre><code>import asyncio
  from concurrent.futures import ThreadPoolExecutor
  from multiprocessing import current_process, Process, Queue
  from threading import current_thread
  from time import sleep
  
  def get_q(q):
      print("Waiting for the queue ({} / {})\n".format(current_thread().name, current_process().name))
      return q.get()
  
  async def message_q(q):
      while True:
          f = loop.run_in_executor(None, get_q, q)
  
          await f
  
          if f.result() is None:
              print("Done")
              return;
  
          print("Got the result ({} / {})".format(current_thread().name, current_process().name))
          print("Result is: {}\n".format(f.result()))
  
  async def something_else():
      while True:
          print("Something else\n")
          await asyncio.sleep(2)
  
  def other_process(q):
      for i in range(5):
          print("Putting something in the queue ({})".format(current_process().name))
          q.put(i)
          sleep(1)
  
      q.put(None)
  
  q = Queue()
  
  Process(target=other_process, args=(q,), daemon=True).start()
  
  loop = asyncio.get_event_loop()
  loop.set_default_executor(ThreadPoolExecutor(max_workers=1))
  asyncio.ensure_future(message_q(q))
  asyncio.ensure_future(something_else())
  loop.run_until_complete(asyncio.sleep(6))
  </code></pre>
  
  <p><code>other_process()</code> is an exemplary worker process which uses a Queue to signal the main process which runs an event loop to process stuff and also wait for any data on the Queue. In the real case, this process would signal the event dispatcher which would then handle the queue messaging, to pass the message on to the main process event dispatcher, but here I simplified it a bit.</p>
  
  <p>However, I'm not quite satisfied with this. Submitting <code>get_q()</code> again and again to a <code>ThreadPoolExecutor</code> produces more overhead and isn't as clean as just one long-running thread. Also the <code>await f</code> there isn't optimal and blocks as soon as no further data is in the queue, which prevents the event loop from exiting. My workaround is to send <code>None</code> after the workers have finished and exit <code>message_q()</code> if <code>None</code> is in the queue.</p>
  
  <p>Is there any better way to implement this? Performance is quite crucial and I would like to keep the Queue object local to the event dispatcher and not pass it to the code that manages the worker processes (or require calling some sort of <code>finalize()</code> method thereof).</p>