title: Python Apache Beam Pipeline Status API Call
body: <p>We currently have a Python Apache Beam pipeline working and able to be run locally. We are now in the process of having the pipeline run on Google Cloud Dataflow and be fully automated but have a found a limitation in Dataflow/Apache Beam's pipeline monitoring.</p>
  
  <p>Currently, <a href="https://cloud.google.com/dataflow/pipelines/dataflow-monitoring-intf" rel="nofollow noreferrer">Cloud Dataflow</a> has two ways of monitoring your pipeline(s) status, either through their UI interface or through gcloud in the command line. Both of these solutions do not work great for a fully automated solution where we can account for loss-less file processing.</p>
  
  <p>Looking at Apache Beam's github they have a file, <a href="https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/internal/apiclient.py" rel="nofollow noreferrer">internal/apiclient.py</a> that shows there is a function used to get the status of a job, <strong>get_job</strong>.</p>
  
  <p>The one instance that we have found get_job used is in <a href="https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/runners/dataflow_runner.py#L105" rel="nofollow noreferrer">runners/dataflow_runner.py</a>.</p>
  
  <p>The end goal is to use this API to get the status of a job or several jobs that we automatically trigger to run to ensure they are all eventually processed successfully through the pipeline.</p>
  
  <p>Can anyone explain to us how this API can be used after we run our pipeline (<code>p.run()</code>)? We do not understand where <code>runner</code> in <code>response = runner.dataflow_client.get_job(job_id)</code> comes from. </p>
  
  <p>If someone could provide a larger understanding of how we can access this API call while setting up / running our pipeline that would be great!</p>