title: Using asyncio to wait for results from subprocess
body: <p>My Python script contains a loop that uses <code>subprocess</code> to run commands outside the script. Each subprocess is independent. I listen for the returned message in case there's an error; I can't ignore the result of the subprocess. Here's the script without asyncio (I've replaced my computationally expensive call with <code>sleep</code>):</p>
  <pre><code>from subprocess import PIPE  # https://docs.python.org/3/library/subprocess.html
  import subprocess
  
  def go_do_something(index: int) -&gt; None:
      &quot;&quot;&quot;
      This function takes a long time
      Nothing is returned
      Each instance is independent
      &quot;&quot;&quot;
      process = subprocess.run([&quot;sleep&quot;,&quot;2&quot;],stdout=PIPE,stderr=PIPE,timeout=20)
      stdout = process.stdout.decode(&quot;utf-8&quot;)
      stderr = process.stderr.decode(&quot;utf-8&quot;)
      if &quot;error&quot; in stderr:
          print(&quot;error for &quot;+str(index))
      return
  
  def my_long_func(val: int) -&gt; None:
      &quot;&quot;&quot;
      This function contains a loop
      Each iteration of the loop calls a function
      Nothing is returned
      &quot;&quot;&quot;
      for index in range(val):
          print(&quot;index = &quot;+str(index))
          go_do_something(index)
  
  # run the script
  my_long_func(3) # launch three tasks
  </code></pre>
  <p>I think I could use <code>asyncio</code> to speed up this activity since the Python script is waiting on the external <code>subprocess</code> to complete. I think <code>threading</code> or <code>multiprocessing</code> are not necessary, though they could also result in faster execution. Using a task queue (e.g., Celery) is another option.</p>
  <p>I tried implementing the <code>asyncio</code> approach, but am missing something since the following attempt doesn't change the overall execution time:</p>
  <pre><code>import asyncio
  from subprocess import PIPE  # https://docs.python.org/3/library/subprocess.html
  import subprocess
  
  
  async def go_do_something(index: int) -&gt; None:
      &quot;&quot;&quot;
      This function takes a long time
      Nothing is returned
      Each instance is independent
      &quot;&quot;&quot;
      process = subprocess.run([&quot;sleep&quot;,&quot;2&quot;],stdout=PIPE,stderr=PIPE,timeout=20)
      stdout = process.stdout.decode(&quot;utf-8&quot;)
      stderr = process.stderr.decode(&quot;utf-8&quot;)
      if &quot;error&quot; in stderr:
          print(&quot;error for &quot;+str(index))
      return
  
  def my_long_func(val: int) -&gt; None:
      &quot;&quot;&quot;
      This function contains a loop
      Each iteration of the loop calls a function
      Nothing is returned
      &quot;&quot;&quot;
      # https://docs.python.org/3/library/asyncio-eventloop.html
      loop = asyncio.get_event_loop()
      tasks = []
      for index in range(val):
          task = go_do_something(index)
          tasks.append(task)
      # https://docs.python.org/3/library/asyncio-task.html
      tasks = asyncio.gather(*tasks)
      loop.run_until_complete(tasks)
      loop.close()
      return
  
  my_long_func(3) # launch three tasks
  </code></pre>
  <p>If I want to monitor the output of each <code>subprocess</code> but not wait while each <code>subprocess</code> runs, can I benefit from <code>asyncio</code>? Or does this situation require something like <code>multiprocessing</code> or Celery?</p>