title: python request urls parallel
body: <p>I created the following script to download images from an API endpoint which works as intended. Thing is that it is rather slow as all the requests have to wait on each other. What is the correct way to make it possible to still have the steps synchronously for each item I want to fetch, but make it parallel for each individual item. This from an online service called
  <a href="http://developer.servicem8.com/docs/rest-api/overview/" rel="nofollow">servicem8</a>
  So what I hope to achieve is:</p>
  
  <ul>
  <li>fetch all possible job ids => keep name/and other info</li>
  <li>fetch name of the customer</li>
  <li>fetch each attachment of a job</li>
  </ul>
  
  <p>These three steps should be done for each job. So I could make things parallel for each job as they do not have to wait on each other.</p>
  
  <p><strong>Update</strong>:</p>
  
  <p>Problem I do not understand is how can you make sure that you bundle for example the three calls per item in one call as its only per item that I can do things in parallel so for example when I want to </p>
  
  <ul>
  <li>fetch item( fetch name => fetch description => fetch id) </li>
  </ul>
  
  <p>so its the fetch item I want to make parallel?</p>
  
  <p>The current code I have is working but rather slow:</p>
  
  <pre><code>import requests
  import dateutil.parser
  import shutil
  import os
  
  user = "test@test.com"
  passw = "test"
  
  print("Read json")
  url = "https://api.servicem8.com/api_1.0/job.json"
  r = requests.get(url, auth=(user, passw))
  
  print("finished reading jobs.json file")
  scheduled_jobs = []
  if r.status_code == 200:
      for item in r.json():
          scheduled_date = item['job_is_scheduled_until_stamp']
          try:
              parsed_date = dateutil.parser.parse(scheduled_date)
              if parsed_date.year == 2016:
                  if parsed_date.month == 10:
                      if parsed_date.day == 10:
                          url_customer = "https://api.servicem8.com/api_1.0/Company/{}.json".format(item[
                                                                                                    'company_uuid'])
                          c = requests.get(url_customer, auth=(user, passw))
                          cus_name = c.json()['name']
                          scheduled_jobs.append(
                              [item['uuid'], item['generated_job_id'], cus_name])
  
          except ValueError:
              pass
  
      for job in scheduled_jobs:
          print("fetch for job {}".format(job))
          url = "https://api.servicem8.com/api_1.0/Attachment.json?%24filter=related_object_uuid%20eq%20{}".format(job[
                                                                                                                   0])
          r = requests.get(url, auth=(user, passw))
          if r.json() == []:
              pass
          for attachment in r.json():
              if attachment['active'] == 1 and attachment['file_type'] != '.pdf':
                  print("fetch for attachment {}".format(attachment))
                  url_staff = "https://api.servicem8.com/api_1.0/Staff.json?%24filter=uuid%20eq%20{}".format(
                      attachment['created_by_staff_uuid'])
                  s = requests.get(url_staff, auth=(user, passw))
                  for staff in s.json():
                      tech = "{}_{}".format(staff['first'], staff['last'])
  
                  url = "https://api.servicem8.com/api_1.0/Attachment/{}.file".format(attachment[
                                                                                      'uuid'])
                  r = requests.get(url, auth=(user, passw), stream=True)
                  if r.status_code == 200:
                      creation_date = dateutil.parser.parse(
                          attachment['timestamp']).strftime("%d.%m.%y")
                      if not os.path.exists(os.getcwd() + "/{}/{}".format(job[2], job[1])):
                          os.makedirs(os.getcwd() + "/{}/{}".format(job[2], job[1]))
                      path = os.getcwd() + "/{}/{}/SC -O {} {}{}".format(
                          job[2], job[1], creation_date, tech.upper(), attachment['file_type'])
                      print("writing file to path {}".format(path))
                      with open(path, 'wb') as f:
                          r.raw.decode_content = True
                          shutil.copyfileobj(r.raw, f)
  else:
      print(r.text)
  </code></pre>
  
  <p><strong>Update [14/10]</strong>
  I updated the code in the following way with some hints given. Thanks a lot for that. Only thing I could optimize I guess is the attachment downloading but it is working fine now. Funny thing I learned is that you cannot create a CON folder on a windows machine :-) did not know that.</p>
  
  <p>I use pandas as well just to try to avoid some loops in my list of dicts but not sure if I am already most performant. Longest is actually reading in the full json files. I fully read them in as I could not find an API way of just telling the api, return me only the jobs from september 2016. The api query function seems to work on eq/lt/ht.</p>
  
  <pre><code>import requests
  import dateutil.parser
  import shutil
  import os
  import pandas as pd
  
  user = ""
  passw = ""
  
  FOLDER = os.getcwd()
  headers = {"Accept-Encoding": "gzip, deflate"}
  
  import grequests
  urls = [
      'https://api.servicem8.com/api_1.0/job.json',
      'https://api.servicem8.com/api_1.0/Attachment.json',
      'https://api.servicem8.com/api_1.0/Staff.json',
      'https://api.servicem8.com/api_1.0/Company.json'
  ]
  
  #Create a set of unsent Requests:
  
  print("Read json files")
  rs = (grequests.get(u, auth=(user, passw), headers=headers) for u in urls)
  #Send them all at the same time:
  jobs,attachments,staffs,companies = grequests.map(rs)
  
  #create dataframes
  df_jobs = pd.DataFrame(jobs.json())
  df_attachments = pd.DataFrame(attachments.json())
  df_staffs = pd.DataFrame(staffs.json())
  df_companies = pd.DataFrame(companies.json())
  
  #url_customer = "https://api.servicem8.com/api_1.0/Company/{}.json".format(item['company_uuid'])
  #c = requests.get(url_customer, auth=(user, passw))
  
  #url = "https://api.servicem8.com/api_1.0/job.json"
  #jobs = requests.get(url, auth=(user, passw), headers=headers)
  
  
  #print("Reading attachments json")
  #url = "https://api.servicem8.com/api_1.0/Attachment.json"
  #attachments = requests.get(url, auth=(user, passw), headers=headers)
  
  #print("Reading staff.json")
  #url_staff = "https://api.servicem8.com/api_1.0/Staff.json"
  #staffs = requests.get(url_staff, auth=(user, passw))
  
  scheduled_jobs = []
  
  if jobs.status_code == 200:
      print("finished reading json file")
      for job in jobs.json():
          scheduled_date = job['job_is_scheduled_until_stamp']
          try:
              parsed_date = dateutil.parser.parse(scheduled_date)
              if parsed_date.year == 2016:
                  if parsed_date.month == 9:
                      cus_name = df_companies[df_companies.uuid == job['company_uuid']].iloc[0]['name'].upper()
                      cus_name = cus_name.replace('/', '')
                      scheduled_jobs.append([job['uuid'], job['generated_job_id'], cus_name])
  
          except ValueError:
              pass
      print("{} jobs to fetch".format(len(scheduled_jobs)))
  
      for job in scheduled_jobs:
          print("fetch for job attachments {}".format(job))
          #url = "https://api.servicem8.com/api_1.0/Attachment.json?%24filter=related_object_uuid%20eq%20{}".format(job[0])
  
          if attachments == []:
              pass
          for attachment in attachments.json():
              if attachment['related_object_uuid'] == job[0]:
                  if attachment['active'] == 1 and attachment['file_type'] != '.pdf' and attachment['attachment_source'] != 'INVOICE_SIGNOFF':
                      for staff in staffs.json():
                          if staff['uuid'] == attachment['created_by_staff_uuid']:
                              tech = "{}_{}".format(
                                  staff['first'].split()[-1].strip(), staff['last'])
  
                      creation_timestamp = dateutil.parser.parse(
                          attachment['timestamp'])
                      creation_date = creation_timestamp.strftime("%d.%m.%y")
                      creation_time = creation_timestamp.strftime("%H_%M_%S")
  
                      path = FOLDER + "/{}/{}/SC_-O_D{}_T{}_{}{}".format(
                          job[2], job[1], creation_date, creation_time, tech.upper(), attachment['file_type'])
  
                      # fetch attachment
  
                      if not os.path.isfile(path):
                          url = "https://api.servicem8.com/api_1.0/Attachment/{}.file".format(attachment[
                                                                                              'uuid'])
                          r = requests.get(url, auth=(user, passw), stream = True)
                          if r.status_code == 200:
                              if not os.path.exists(FOLDER + "/{}/{}".format(job[2], job[1])):
                                  os.makedirs(
                                      FOLDER + "/{}/{}".format(job[2], job[1]))
  
                              print("writing file to path {}".format(path))
                              with open(path, 'wb') as f:
                                  r.raw.decode_content = True
                                  shutil.copyfileobj(r.raw, f)
                      else:
                          print("file already exists")
  else:
      print(r.text)
  </code></pre>