title: How to speed up downloading data from Quandl/SHARADAR api
body: <p>I've built a small download manager to get data for the SHARADAR tables in Quandl. <a href="https://github.com/neilsmurphy/sharadar-download-manager" rel="nofollow noreferrer">GIT</a></p>
  
  <p>This is functioning well but the downloads are very slow for the larger files (up to 2 gb over 10 years). </p>
  
  <p>I attempted to use asyncio but this didn't speed up the downloads. This may be because <a href="https://blog.quandl.com/change-quandl-api-limits" rel="nofollow noreferrer">Quandl doesn't allow concurrent downloads</a>. Am I making an error in my code, or is this restriction I will have to live with from Quandl?</p>
  
  <pre><code>import asyncio
  import math
  import time
  
  import pandas as pd
  import quandl
  
  import update
  
  def segment_dates(table, date_start, date_end):
  
      # Determine the number of days per asyncio loop. Determined by the max size of the
      # range of data divided by the size of the files in 100 mb chunks.
      # reduce this number for smaller more frequent downloads.
      total_days = 40
  
      # Number of days per download should be:
      sizer = math.ceil(total_days / update.sharadar_tables[table][2])
  
      # Number of days between start and end.
      date_diff = date_end - date_start
  
      loop_count = int(math.ceil(date_diff.days / sizer))
      sd = date_start
      sync_li = []
      for _ in range(loop_count):
          ed = sd + pd.Timedelta(days=sizer)
          if ed &gt; date_end:
              ed = date_end
          sync_li.append((sd, ed,))
          sd = ed + pd.Timedelta(days=1)
  
      return sync_li
  
  
  async def get_data(table, kwarg):
      """
      Using the table name and kwargs retrieves the most current data.
      :param table: Name of table to update.
      :param kwarg: Dictionary containing the parameters to send to Quandl.
      :return dataframe: Pandas dataframe containing latest data for the table.
      """
      return quandl.get_table("SHARADAR/" + table.upper(), paginate=True, **kwarg)
  
  
  async def main():
  
      table = "SF1"
  
      # Name of the column that has the date field for this particular table.
      date_col = update.sharadar_tables[table][0]
  
      date_start = pd.to_datetime("2020-03-15")
      date_end = pd.to_datetime("2020-04-01")
  
      apikey = "API Key"
      quandl.ApiConfig.api_key = apikey
  
      # Get a list containing the times start and end for loops.
      times = segment_dates(table, date_start, date_end)
  
      wait_li = []
      for t in times:
          kwarg = {date_col: {"gte": t[0].strftime("%Y-%m-%d"), "lte": t[1].strftime("%Y-%m-%d")}}
          wait_li.append(loop.create_task(get_data(table, kwarg)))
  
      await asyncio.wait(wait_li)
      return wait_li
  
  if __name__ == "__main__":
      starter = time.time()
      try:
          loop = asyncio.get_event_loop()
          res = loop.run_until_complete(main())
          for r in res:
              df = r.result()
              print(df.shape)
              print(df.head())
      except:
          raise ValueError("error")
      finally:
          # loop.close()
  
      print("Finished in {}".format(time.time() - starter))
  
  </code></pre>