<p>I'm submitting Spark jobs in Kubernetes running locally (Docker desktop). I'm able to submit the jobs and see their final output in the screen.</p>

<p>However, even if they're completed, the driver and executor pods are still in a RUNNING state.</p>

<p>The base images used to submit the Spark jobs to kubernetes are the ones that come with Spark, as described in the <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">docs</a>.</p>

<p>This is what my <code>spark-submit</code> command looks like:</p>

<pre><code>~/spark-2.4.3-bin-hadoop2.7/bin/spark-submit \
    --master k8s://https://kubernetes.docker.internal:6443 \
    --deploy-mode cluster \
    --name my-spark-job \
    --conf spark.kubernetes.container.image=my-spark-job \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.submission.waitAppCompletion=false \
    local:///opt/spark/work-dir/my-spark-job.py
</code></pre>

<p>And this is what <code>kubectl get pods</code> returns:</p>

<pre><code>NAME                                READY   STATUS    RESTARTS   AGE
my-spark-job-1568669908677-driver   1/1     Running   0          11m
my-spark-job-1568669908677-exec-1   1/1     Running   0          10m
my-spark-job-1568669908677-exec-2   1/1     Running   0          10m
</code></pre>

<p>Figured it out. I forgot to <code>stop</code> the Spark Context. My script looks like this now, and at completion, the driver goes into <code>Completed</code> status and the drivers get deleted.</p>

<pre><code>sc = SparkContext()

sqlContext = SQLContext(sc)

# code

sc.stop()
</code></pre>
