title: Using asyncio.Queue for producer-consumer flow
body: <p>I'm confused about how to use <code>asyncio.Queue</code> for a particular producer-consumer pattern in which both the producer and consumer operate concurrently and independently.</p>
  
  <p>First, consider this example, which closely follows that from the <a href="https://docs.python.org/3/library/asyncio-queue.html#examples" rel="noreferrer">docs for <code>asyncio.Queue</code></a>:</p>
  
  <pre><code>import asyncio
  import random
  import time
  
  async def worker(name, queue):
      while True:
          sleep_for = await queue.get()
          await asyncio.sleep(sleep_for)
          queue.task_done()
          print(f'{name} has slept for {sleep_for:0.2f} seconds')
  
  async def main(n):
      queue = asyncio.Queue()
      total_sleep_time = 0
      for _ in range(20):
          sleep_for = random.uniform(0.05, 1.0)
          total_sleep_time += sleep_for
          queue.put_nowait(sleep_for)
      tasks = []
      for i in range(n):
          task = asyncio.create_task(worker(f'worker-{i}', queue))
          tasks.append(task)
      started_at = time.monotonic()
      await queue.join()
      total_slept_for = time.monotonic() - started_at
      for task in tasks:
          task.cancel()
      # Wait until all worker tasks are cancelled.
      await asyncio.gather(*tasks, return_exceptions=True)
      print('====')
      print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds')
      print(f'total expected sleep time: {total_sleep_time:.2f} seconds')
  
  if __name__ == '__main__':
      import sys
      n = 3 if len(sys.argv) == 1 else sys.argv[1]
      asyncio.run(main())
  </code></pre>
  
  <p>There is one finer detail about this script: the items are put into the queue synchronously, with <code>queue.put_nowait(sleep_for)</code> over a conventional for-loop.</p>
  
  <p>My goal is to create a script that uses <code>async def worker()</code> (or <code>consumer()</code>) and <code>async def producer()</code>.  Both should be scheduled to run concurrently.  No one consumer coroutine is explicitly tied to or chained from a producer.</p>
  
  <p>How can I modify the program above so that the producer(s) is its own coroutine that can be scheduled concurrently with the consumers/workers?</p>
  
  <hr>
  
  <p><sub>There is a second example from <a href="https://pymotw.com/3/asyncio/synchronization.html#queues" rel="noreferrer">PYMOTW</a>.  It requires the producer to know the number of consumers ahead of time, and uses <code>None</code> as a signal to the consumer that production is done.</sub></p>