title: How paginate through api response asynchronously with asyncio and aiohttp
body: <p>I'm trying to make api calls with python asynchronously. I have multiple endpoints in a list and each endpoint will return paginated results. I'm able to set up going though the multiple endpoints asynchronously, however am not able to return the paginated results of each endpoint. </p>
  
  <p>From debugging, I found that the <code>fetch_more()</code> function runs the while loop, but doesn't actually get past the async with session.get(). So basically. The function <code>fetch_more()</code> is intended to get remaining results from the api call for each endpoint, however I find that the same number of results are returned with or without the <code>fetch_more()</code> function. I've tried looking for examples of pagination with asyncio but have not have much luck. </p>
  
  <p>From my understanding, I should not be doing a request inside a while loop, however, I'm not sure a way around that in order to get paginated results. </p>
  
  <pre><code>if __name__ == 'main':
  
      starter_fun(url, header, endpoints):
  
  starter_func(url, header, endpoints):
  
      loop = asyncio.get_event_loop() #event loop
      future = asyncio.ensure_future(fetch_all(url, header, endpoints))
      loop.run_until_complete(future) #loop until done
  
  async def fetch_all(url, header, endpoints):
  
      async with ClientSession() as session:
          for endpoint in endpoints:
             task = asyncio.ensure_future(fetch(url, header, endpoint))
             tasks.append(task)
          res = await asyncio.gather(*tasks) # gather task responses
          return res
  
  async def fetch(url, header, endpoint): 
  
      total_tasks = []
      async with session.get(url, headers=header, params=params, ssl=False) as response:
          response_json = await response.json()
          data = response_json['key']
         tasks = asyncio.ensure_future(fetch_more(response_json, data, params, header, url, endpoint, session)) //this is where I am getting stuck
          total_tasks.append(tasks)
      return data
  
  
  //function to get paginated results of api endpoint
  async def fetch_more(response_json, data, params, header, url, endpoint, session): //this is where I am getting stuck
  
      while len(response_json['key']) &gt;= params['limit']:
          params['offset'] = response_json['offset'] + len(response_json['key'])
          async with session.get(url, headers=header, params=params, ssl=False) as response_continued:
              resp_continued_json = await response_continued.json()
              data.extend(resp_continued_json[kebab_to_camel(endpoint)])
     return data
  
  </code></pre>
  
  <p>Currently I am getting 1000 results with or without the <code>fetch_more</code> function, however it should be a lot more with the <code>fetch_more</code>. Any idea as to how to approach asynchronously paginating?</p>