<p>How to check if the spark job is succeeded or failed programmatically, while running spark-submit. Usually the unix exit code is used.</p>
<pre><code> phase: Failed
 container status:
     container name: spark-kubernetes-driver
     container image: &lt;regstry&gt;/spark-py:spark3.2.1
     container state: terminated
     container started at: 2022-03-25T19:10:51Z
     container finished at: 2022-03-25T19:10:57Z
     exit code: 1
     termination reason: Error

2022-03-25 15:10:58,457 INFO submit.LoggingPodStatusWatcherImpl: Application Postgres-Minio-Kubernetes.py with submission ID spark:postgres-minio-kubernetes-py-b70d3f7fc27829ec-driver finished
2022-03-25 15:10:58,465 INFO util.ShutdownHookManager: Shutdown hook called
2022-03-25 15:10:58,466 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3321e67c-73d5-422d-a26d-642a0235cf23
</code></pre>
<p>Process failed and when I get the exit code in unix by echo $? it return a zero error code!</p>
<pre><code>$ echo $?
0
</code></pre>
<p>The pod generated is also random way. What's the way the spark-submit is handled apart from using sparkonk8operator?</p>

Rafa
<p>If you are using bash, one way to to grep on the output. You might have to grep on <a href="https://en.wikipedia.org/wiki/Standard_streams" rel="nofollow noreferrer"><code>stderr</code> or <code>stdout</code></a> depending on where the log output is being sent.</p>
<p>Something like this:</p>
<pre><code>OUTPUT=`spark-submit ...`
if echo &quot;$OUTPUT&quot; | grep -q &quot;exit code: 1&quot;; then
    exit 1
fi
</code></pre>
