title: Controlling the concurrency of HTTP requests using Python's asyncio.Semaphore
body: <p>I'm trying to figure out a way to limit the number of concurrent HTTP requests made to a server using Python's <strong>asyncio</strong> and <strong>httpx</strong> module.  <strong>I came across this StackOverflow <a href="https://stackoverflow.com/a/48486557/8963300">answer</a>.</strong></p>
  <p>It proposes <code>asyncio.Semaphore</code> for stopping multiple consumers from making too many requests. While this answer works perfectly, it uses explicit loop construction, not <code>asyncio.run</code>. When I replace the explicit loop construction with <code>asyncio.run</code>, the behavior of the code changes. Instead of doing all 9 requests, now it just executes three requests and then stops.</p>
  <pre><code>import asyncio
  from random import randint
  
  
  async def download(code):
      wait_time = randint(1, 3)
      print('downloading {} will take {} second(s)'.format(code, wait_time))
      await asyncio.sleep(wait_time)  # I/O, context will switch to main function
      print('downloaded {}'.format(code))
  
  
  sem = asyncio.Semaphore(3)
  
  
  async def safe_download(i):
      async with sem:  # semaphore limits num of simultaneous downloads
          return await download(i)
  
  
  async def main():
      tasks = [
          asyncio.ensure_future(safe_download(i))  # creating task starts coroutine
          for i
          in range(9)
      ]
      await asyncio.gather(*tasks, return_exceptions=True)  # await moment all downloads done
  
  
  if __name__ ==  '__main__':
      asyncio.run(main())
  </code></pre>
  <p>This prints out:</p>
  <pre><code>downloading 0 will take 3 second(s)
  downloading 1 will take 1 second(s)
  downloading 2 will take 3 second(s)
  downloaded 1
  downloaded 0
  downloaded 2
  </code></pre>
  <p>I had to change <code>await asyncio.gather(*tasks)</code> to <code>await asyncio.gather(*tasks, return_exceptions=True)</code> so that the code doesn't throw a <code>RuntimeError</code>. Otherwise it'd throw this error, I've got asyncio debug mode turned on.</p>
  <pre><code>downloading 0 will take 2 second(s)
  downloading 1 will take 3 second(s)
  downloading 2 will take 1 second(s)
  Traceback (most recent call last):
    File &quot;/home/rednafi/workspace/personal/demo/demo.py&quot;, line 66, in &lt;module&gt;
      asyncio.run(main())
    File &quot;/usr/lib/python3.9/asyncio/runners.py&quot;, line 44, in run
      return loop.run_until_complete(main)
    File &quot;/usr/lib/python3.9/asyncio/base_events.py&quot;, line 642, in run_until_complete
      return future.result()
    File &quot;/home/rednafi/workspace/personal/demo/demo.py&quot;, line 62, in main
      await asyncio.gather(*tasks)  # await moment all downloads done
    File &quot;/home/rednafi/workspace/personal/demo/demo.py&quot;, line 52, in safe_download
      async with sem:  # semaphore limits num of simultaneous downloads
    File &quot;/usr/lib/python3.9/asyncio/locks.py&quot;, line 14, in __aenter__
      await self.acquire()
    File &quot;/usr/lib/python3.9/asyncio/locks.py&quot;, line 413, in acquire
      await fut
  RuntimeError: Task &lt;Task pending name='Task-5' coro=&lt;safe_download() running at /home/rednafi/workspace/personal/demo/demo.py:52&gt; cb=[gather.&lt;locals&gt;._done_callback() at /usr/lib/python3.9/asyncio/tasks.py:764] created at /home/rednafi/workspace/personal/demo/demo.py:58&gt; got Future &lt;Future pending created at /usr/lib/python3.9/asyncio/base_events.py:424&gt; attached to a different loop
  
  </code></pre>
  <p>However, the only other change is replacing the explicit loop with <code>asyncio.run</code>.</p>
  <p>The question is why the behavior of the code changed? And how can I bring back the old, expected behavior?</p>