<p>As far as I know, when most people want to know if a <code>Kubernetes</code> (or <code>Spark</code> even) <code>Job</code> is done, they initiate some sort of loop somewhere to periodically check if the <code>Job</code> is finished with the respective <code>API</code>.</p>

<p>Right now, I'm doing that with <code>Kubernetes</code> in the background with the <code>disown</code> (<code>&amp;</code>) operator (<code>bash</code> inside <code>Python</code> below):</p>

<pre class="lang-py prettyprint-override"><code>import subprocess

cmd = f'''
kubectl wait \\
    --for=condition=complete \\
    --timeout=-1s \\
    job/job_name \\
    &gt; logs/kube_wait_log.txt \\
    &amp;
'''

kube_listen = subprocess.run(
    cmd,
    shell = True,
    stdout = subprocess.PIPE
)
</code></pre>

<p>So... I actually have <em>two</em> (correlated) questions:</p>

<ol>
<li>Is there a better way of doing this in the background with <code>shell</code> other than with the <code>&amp;</code> operator?</li>
<li>The option that I think would be best is actually to use <code>cURL</code> from inside the <code>Job</code> to update my <code>Local Server API</code> that interacts with <code>Kubernetes</code>. 

<ul>
<li>However, I don't know how I can perform a <code>cURL</code> from a <code>Job</code>. Is it possible?</li>
<li>I imagine you would have to expose ports somewhere but where? And is it really supported? Could you create a <code>Kubernetes</code> <code>Service</code> to manage the ports and connections? </li>
</ul></li>
</ol>

Philippe Fanaro
<p>If you don't want to block on a process running to completion, you can create a <a href="https://docs.python.org/3/library/subprocess.html#subprocess.Popen" rel="nofollow noreferrer"><code>subprocess.Popen</code></a> instance instead.  Once you have this, you can <a href="https://docs.python.org/3/library/subprocess.html#subprocess.Popen.poll" rel="nofollow noreferrer"><code>poll()</code></a> it to see if it's completed.  (You should try really really really hard to avoid using <code>shell=True</code> if at all possible.)  So one variation of this could look like (untested):</p>

<pre class="lang-py prettyprint-override"><code>with open('logs/kube_wait_log.txt', 'w') as f:
  with subprocess.Popen(['kubectl', 'wait',
                         '--for=condition=complete',
                         '--timeout=-1s',
                         'job/job_name'],
                         stdin=subprocess.DEVNULL,
                         stdout=f,
                         stderr=subprocess.STDOUT) as p:
    while True:
      if p.poll():
        job_is_complete()
        break
      time.sleep(1)
</code></pre>

<p>Better than shelling out to <code>kubectl</code>, though, is using the <a href="https://github.com/kubernetes-client/python/" rel="nofollow noreferrer">official Kubernetes Python client library</a>.  Rather than using this "wait" operation, you would <a href="https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes" rel="nofollow noreferrer">watch</a> the job object in question and see if its status changes to "completed".  This could look roughly like (untested):</p>

<pre class="lang-py prettyprint-override"><code>from kubernetes import client, watch
jobsv1 = client.BatchV1Api()
w = watch.watch()
for event in w.stream(jobsv1.read_namespaced_job, 'job_name', 'default'):
  job = event['object']
  if job.status.completion_time is not None:
    job_is_complete()
    break
</code></pre>

<p>The Job's Pod doesn't need to update its own status with the Kubernetes server.  It just needs to exit with a successful status code (0) when it's done, and that will get reflected in the Job's status field.</p>
