title: How to limit concurrency with Python asyncio?
body: <p>Let's assume we have a bunch of links to download and each of the link may take a different amount of time to download. And I'm allowed to download using utmost 3 connections only. Now, I want to ensure that I do this efficiently using asyncio.</p>
  <p>Here's what I'm trying to achieve: At any point in time, try to ensure that I have atleast 3 downloads running.</p>
  <pre><code>Connection 1: 1---------7---9---
  Connection 2: 2---4----6-----
  Connection 3: 3-----5---8-----
  </code></pre>
  <p>The numbers represent the download links, while hyphens represent Waiting for download.</p>
  <p>Here is the code that I'm using right now</p>
  <pre><code>from random import randint
  import asyncio
  
  count = 0
  
  
  async def download(code, permit_download, no_concurrent, downloading_event):
      global count
      downloading_event.set()
      wait_time = randint(1, 3)
      print('downloading {} will take {} second(s)'.format(code, wait_time))
      await asyncio.sleep(wait_time)  # I/O, context will switch to main function
      print('downloaded {}'.format(code))
      count -= 1
      if count &lt; no_concurrent and not permit_download.is_set():
          permit_download.set()
  
  
  async def main(loop):
      global count
      permit_download = asyncio.Event()
      permit_download.set()
      downloading_event = asyncio.Event()
      no_concurrent = 3
      i = 0
      while i &lt; 9:
          if permit_download.is_set():
              count += 1
              if count &gt;= no_concurrent:
                  permit_download.clear()
              loop.create_task(download(i, permit_download, no_concurrent, downloading_event))
              await downloading_event.wait()  # To force context to switch to download function
              downloading_event.clear()
              i += 1
          else:
              await permit_download.wait()
      await asyncio.sleep(9)
  
  if __name__ == '__main__':
      loop = asyncio.get_event_loop()
      try:
          loop.run_until_complete(main(loop))
      finally:
          loop.close()
  </code></pre>
  <p>And the output is as expected:</p>
  <pre><code>downloading 0 will take 2 second(s)
  downloading 1 will take 3 second(s)
  downloading 2 will take 1 second(s)
  downloaded 2
  downloading 3 will take 2 second(s)
  downloaded 0
  downloading 4 will take 3 second(s)
  downloaded 1
  downloaded 3
  downloading 5 will take 2 second(s)
  downloading 6 will take 2 second(s)
  downloaded 5
  downloaded 6
  downloaded 4
  downloading 7 will take 1 second(s)
  downloading 8 will take 1 second(s)
  downloaded 7
  downloaded 8
  </code></pre>
  <p>But here are my questions:</p>
  <ol>
  <li><p>At the moment, I'm simply waiting for 9 seconds to keep the main function running till the downloads are complete. Is there an efficient way of waiting for the last download to complete before exiting the <code>main</code> function? (I know there's <code>asyncio.wait</code>, but I'll need to store all the task references for it to work)</p>
  </li>
  <li><p>What's a good library that does this kind of task? I know javascript has a lot of async libraries, but what about Python?</p>
  </li>
  </ol>
  <p>Edit:
  2. What's a good library that takes care of common async patterns? (Something like <a href="https://www.npmjs.com/package/async" rel="noreferrer">async</a>)</p>