title: Spark job in Kubernetes stuck in RUNNING state
body: <p>I'm submitting Spark jobs in Kubernetes running locally (Docker desktop). I'm able to submit the jobs and see their final output in the screen.</p>
  
  <p>However, even if they're completed, the driver and executor pods are still in a RUNNING state.</p>
  
  <p>The base images used to submit the Spark jobs to kubernetes are the ones that come with Spark, as described in the <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">docs</a>.</p>
  
  <p>This is what my <code>spark-submit</code> command looks like:</p>
  
  <pre><code>~/spark-2.4.3-bin-hadoop2.7/bin/spark-submit \
      --master k8s://https://kubernetes.docker.internal:6443 \
      --deploy-mode cluster \
      --name my-spark-job \
      --conf spark.kubernetes.container.image=my-spark-job \
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
      --conf spark.kubernetes.submission.waitAppCompletion=false \
      local:///opt/spark/work-dir/my-spark-job.py
  </code></pre>
  
  <p>And this is what <code>kubectl get pods</code> returns:</p>
  
  <pre><code>NAME                                READY   STATUS    RESTARTS   AGE
  my-spark-job-1568669908677-driver   1/1     Running   0          11m
  my-spark-job-1568669908677-exec-1   1/1     Running   0          10m
  my-spark-job-1568669908677-exec-2   1/1     Running   0          10m
  </code></pre>