title: Creating a cluster before sending a job to dataproc programmatically
body: <p>I'm trying to schedule a PySpark Job. I followed the GCP documentation and ended up deploying a little python script to App Engine which does the following : </p>
  
  <ul>
  <li>authenticate using a service account</li>
  <li>submit a job to a cluster</li>
  </ul>
  
  <p>The problem is, I need the cluster to be up and running otherwise the job won't be sent (duh !) but I don't want the cluster to always be up and running, especially since my job needs to run once a month.</p>
  
  <p>I wanted to add the creation of a cluster in my python script but the call is asynchronous (it makes an HTTP request) and thus my job is submitted after the cluster creation call but before the cluster is really up and running.</p>
  
  <p>How could I do ? </p>
  
  <p>I'd like something cleaner than just waiting for a few minutes in my script !</p>
  
  <p>Thanks</p>
  
  <p>EDIT : Here's what my code looks like so far :</p>
  
  <p>To launch the job </p>
  
  <pre><code>class EnqueueTaskHandler(webapp2.RequestHandler):
      def get(self):
          task = taskqueue.add(
              url='/run',
              target='worker')
  
          self.response.write(
              'Task {} enqueued, ETA {}.'.format(task.name, task.eta))
  
  app = webapp2.WSGIApplication([('/launch', EnqueueTaskHandler)], debug=True)
  </code></pre>
  
  <p>The job </p>
  
  <pre><code>class CronEventHandler(webapp2.RequestHandler):
  
      def create_cluster(self, dataproc, project, zone, region, cluster_name):
          zone_uri = 'https://www.googleapis.com/compute/v1/projects/{}/zones/{}'.format(project, zone)
          cluster_data = {...}
  
          dataproc.projects().regions().clusters().create(
              projectId=project,
              region=region,
              body=cluster_data).execute()
  
      def wait_for_cluster(self, dataproc, project, region, clustername):
          print('Waiting for cluster to run...')
          while True:
              result = dataproc.projects().regions().clusters().get(
              projectId=project,
              region=region,
              clusterName=clustername).execute()
              # Handle exceptions
              if result['status']['state'] != 'RUNNING':
                  time.sleep(60)
              else:
                  return result
  
      def wait_for_job(self, dataproc, project, region, job_id):
          print('Waiting for job to finish...')
          while True:
              result = dataproc.projects().regions().jobs().get(
                  projectId=project,
                  region=region,
                  jobId=job_id).execute()
              # Handle exceptions
              print(result['status']['state'])
              if result['status']['state'] == 'ERROR' or result['status']['state'] == 'DONE':
                  return result
              else:
                  time.sleep(60)
  
      def submit_job(self, dataproc, project, region, clusterName):
          job = {...}
          result = dataproc.projects().regions().jobs().submit(projectId=project,region=region,body=job).execute()
          return result['reference']['jobId']
  
  
      def post(self):
          dataproc = googleapiclient.discovery.build('dataproc', 'v1')
  
          project = '...'
          region = "..."
          zone = "..."
          clusterName = '...'
  
          self.create_cluster(dataproc, project, zone, region, clusterName)
          self.wait_for_cluster(dataproc, project, region, clusterName)
          job_id = self.submit_job(dataproc,project,region,clusterName)
          self.wait_for_job(dataproc,project,region,job_id)
          dataproc.projects().regions().clusters().delete(projectId=project, region=region, clusterName=clusterName).execute()
          self.response.write("JOB SENT")
  
  app = webapp2.WSGIApplication([('/run', CronEventHandler)], debug=True)
  </code></pre>
  
  <p>Everything works until the deletion of the cluster. At this point I get a "DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded." Any idea ? </p>