title: Python 3 asyncio with aioboto3 seems sequential
body: <p>I am porting a simple python 3 script to AWS Lambda.
  The script is simple: it gathers information from a dozen of S3 objects and returns the results.</p>
  <p>The script used <code>multiprocessing.Pool</code> to gather all the files in parallel. Though <code>multiprocessing</code> cannot be used in an AWS Lambda environment since <code>/dev/shm</code> is missing.
  So I thought instead of writing a dirty <code>multiprocessing.Process</code> / <code>multiprocessing.Queue</code> replacement, I would try <code>asyncio</code> instead.</p>
  <p>I am using the latest version of <code>aioboto3</code> (8.0.5) on Python 3.8.</p>
  <p>My problem is that I cannot seem to gain any improvement between a naive sequential download of the files, and an asyncio event loop multiplexing the downloads.</p>
  <p>Here are the two versions of my code.</p>
  <pre class="lang-py prettyprint-override"><code>import sys
  import asyncio
  from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
  
  import boto3
  import aioboto3
  
  BUCKET = 'some-bucket'
  KEYS = [
      'some/key/1',
      [...]
      'some/key/10',
  ]
  
  async def download_aio():
      &quot;&quot;&quot;Concurrent download of all objects from S3&quot;&quot;&quot;
      async with aioboto3.client('s3') as s3:
          objects = [s3.get_object(Bucket=BUCKET, Key=k) for k in KEYS]
          objects = await asyncio.gather(*objects)
          buffers = await asyncio.gather(*[o['Body'].read() for o in objects])
  
  def download():
      &quot;&quot;&quot;Sequentially download all objects from S3&quot;&quot;&quot;
      s3 = boto3.client('s3')
      for key in KEYS:
          object = s3.get_object(Bucket=BUCKET, Key=key)
          object['Body'].read()
  
  def run_sequential():
      download()
  
  def run_concurrent():
      loop = asyncio.get_event_loop()
      #loop.set_default_executor(ProcessPoolExecutor(10))
      #loop.set_default_executor(ThreadPoolExecutor(10))
      loop.run_until_complete(download_aio())
  </code></pre>
  <p>The timing for both <code>run_sequential()</code> and <code>run_concurrent()</code> are quite similar (~3 seconds for a dozen of 10MB files).
  I am convinced the concurrent version is not, for multiple reasons:</p>
  <ul>
  <li>I tried switching to <code>Process/ThreadPoolExecutor</code>, and I the processes/threads spawned for the duration of the function, though they are doing nothing</li>
  <li>The timing between sequential and concurrent is very close to the same, though my network interface is definitely not saturated, and the CPU is not bound either</li>
  <li>The time taken by the concurrent version increases linearly with the number of files.</li>
  </ul>
  <p>I am sure something is missing, but I just can't wrap my head around what.</p>
  <p>Any ideas?</p>