<p>Running Spark on Kubernetes, with each of 3 Spark workers given 8 cores and 8G ram, results in</p>
<pre><code>Executor app-xxx-xx/0 finished with state KILLED exitStatus 143
</code></pre>
<p>seemingly no matter how simple the computation or what flags I pass to <code>spark-submit</code>.</p>
<p>For instance,</p>
<pre class="lang-sh prettyprint-override"><code>kubectl run -n redacted spark-client --rm -it --restart='Never' \
  --image docker.io/bitnami/spark:3.2.0-debian-10-r2 \
  -- run-example \
    --name my-pi-calc-example-2 \
    --master spark://spark-master-svc:7077 \
    --deploy-mode cluster \
    --driver-memory 4g \
    --executor-memory 1g \
    --driver-cores 4 \
    --executor-cores 4 \
    --verbose \
    SparkPi 3
</code></pre>
<p>gives me the following logs on <code>spark-worker-0</code>:</p>
<pre><code>21/11/15 22:07:42 INFO DriverRunner: Launch Command: &quot;/opt/bitnami/java/bin/java&quot; &quot;-cp&quot; &quot;/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*&quot; &quot;-Xmx4096M&quot; &quot;-Dspark.master=spark://spark-master-svc:7077&quot; &quot;-Dspark.driver.cores=4&quot; &quot;-Dspark.driver.supervise=false&quot; &quot;-Dspark.submit.deployMode=cluster&quot; &quot;-Dspark.driver.memory=4g&quot; &quot;-Dspark.executor.memory=4g&quot; &quot;-Dspark.submit.pyFiles=&quot; &quot;-Dspark.jars=file:///opt/bitnami/spark/examples/jars/scopt_2.12-3.7.1.jar,file:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.2.0.jar,file:/opt/bitnami/spark/examples/jars/spark-examples_2.12-3.2.0.jar&quot; &quot;-Dspark.rpc.askTimeout=10s&quot; &quot;-Dspark.app.name=my-pi-calc-example-2&quot; &quot;-Dspark.executor.cores=4&quot; &quot;org.apache.spark.deploy.worker.DriverWrapper&quot; &quot;spark://Worker@xx.xx.19.190:34637&quot; &quot;/opt/bitnami/spark/work/driver-20211115220742-0006/spark-examples_2.12-3.2.0.jar&quot; &quot;org.apache.spark.examples.SparkPi&quot; &quot;3&quot; &quot;--verbose&quot;
21/11/15 22:07:44 INFO Worker: Asked to launch executor app-20211115220744-0006/4 for Spark Pi
21/11/15 22:07:44 INFO SecurityManager: Changing view acls to: spark
21/11/15 22:07:44 INFO SecurityManager: Changing modify acls to: spark
21/11/15 22:07:44 INFO SecurityManager: Changing view acls groups to:
21/11/15 22:07:44 INFO SecurityManager: Changing modify acls groups to:
21/11/15 22:07:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
21/11/15 22:07:44 INFO ExecutorRunner: Launch command: &quot;/opt/bitnami/java/bin/java&quot; &quot;-cp&quot; &quot;/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*&quot; &quot;-Xmx4096M&quot; &quot;-Dspark.driver.port=42013&quot; &quot;-Dspark.rpc.askTimeout=10s&quot; &quot;org.apache.spark.executor.CoarseGrainedExecutorBackend&quot; &quot;--driver-url&quot; &quot;spark://CoarseGrainedScheduler@spark-worker-0.spark-headless.redacted.svc.cluster.local:42013&quot; &quot;--executor-id&quot; &quot;4&quot; &quot;--hostname&quot; &quot;xx.xx.19.190&quot; &quot;--cores&quot; &quot;4&quot; &quot;--app-id&quot; &quot;app-20211115220744-0006&quot; &quot;--worker-url&quot; &quot;spark://Worker@xx.xx.19.190:34637&quot;
21/11/15 22:07:48 INFO Worker: Asked to kill executor app-20211115220744-0006/4
21/11/15 22:07:48 INFO ExecutorRunner: Runner thread for executor app-20211115220744-0006/4 interrupted
21/11/15 22:07:48 INFO ExecutorRunner: Killing process!
21/11/15 22:07:48 INFO Worker: Executor app-20211115220744-0006/4 finished with state KILLED exitStatus 143
21/11/15 22:07:48 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 4
21/11/15 22:07:48 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20211115220744-0006, execId=4)
21/11/15 22:07:48 INFO ExternalShuffleBlockResolver: Application app-20211115220744-0006 removed, cleanupLocalDirs = true
21/11/15 22:07:48 INFO Worker: Cleaning up local directories for application app-20211115220744-0006
21/11/15 22:07:48 INFO Worker: Driver driver-20211115220742-0006 exited successfully
</code></pre>
<p>I can drop, change, or modify the <code>run-example</code> or <code>spark-submit</code> flags. It seemingly has no effect, even for something as simple as <code>SparkPi 3</code>; the executors are killed and exit code 143 with very little information about why they were actually killed.</p>
<p>Resource constraints should not be an issue whatsoever here. This is a Kubernetes cluster of 3 AWS m5.4xlarge worker nodes, 16 vCPu and 64GiB RAM with very little else actually deployed on the. I have not set Kubernetes <code>spec.resources</code> on <code>limits</code> or <code>requests</code>. The Spark cluster is deployed as follows:</p>
<pre class="lang-sh prettyprint-override"><code>argocd app create spark \
    --repo https://charts.bitnami.com/bitnami \
    --helm-chart spark \
    --dest-server https://kubernetes.default.svc \
    --insecure \
    --helm-set 'worker.replicaCount=3' \
    --dest-namespace redacted \
    --revision '5.7.9' \
    --helm-set worker.coreLimit=8 \
    --helm-set worker.memoryLimit=8G \
    --helm-set worker.daemonMemoryLimit=4G \
    --helm-set master.daemonMemoryLimit=4G

argocd app sync spark
</code></pre>
<p>This uses the <a href="https://github.com/bitnami/charts/tree/master/bitnami/spark" rel="nofollow noreferrer">Spark Bitnami Helm chart</a> and ArgoCD/Helm to deploy.</p>
<p>The cluster deploys just fine; I can see, for example, <code>Starting Spark worker xxx.xx.xx.xx:46105 with 8 cores, 8.0 GiB RAM</code> and that all 3 workers have joined.</p>
<p>What am I missing here? How can I better debug this and figure out what the resource constraint is?</p>
<hr />
<p>Funny enough, I can even run SparkPi locally. If I e.g. <code>kubectl exec -it spark-worker-0 -- bash</code>:</p>
<pre class="lang-sh prettyprint-override"><code>$ ./bin/run-example SparkPi 3
...
21/11/15 22:22:09 INFO SparkContext: Running Spark version 3.2.0
...
21/11/15 22:22:11 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 0.634538 s
Pi is roughly 3.1437838126127087
</code></pre>
<p>Then I can add two arguments to run in cluster mode, and boom, executors get killed:</p>
<pre><code>$ ./bin/run-example \
    --master spark://spark-master-svc:7077 \
    --deploy-mode cluster SparkPi
# Executor app-20211115222530-0008/2 finished with state KILLED exitStatus 143
</code></pre>

Brad Solomon
<p>Learned a couple things here. The first is that 143 KILLED does not seem to actually be indicative of failure but rather of executors receiving a signal to shutdown once the job is finished. So, seems draconian when found in logs but is not.</p>
<p>What was confusing me was that I wasn't seeing any &quot;Pi is roughly 3.1475357376786883&quot; text on stdout/stderr. This led me to believe the computation never got that far, which was incorrect.</p>
<p>The issue here is what I was using <code>--deploy-mode cluster</code> when <code>--deploy-mode client</code> actually made a lot more sense in this situation. That is because I was running an ad-hoc container through <code>kubectl run</code> which was not part of the existing deployment. This fits the definition of <a href="https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit" rel="nofollow noreferrer">client mode</a> better, since the submission does not come from an existing Spark worker. When running in <code>--deploy-mode=cluster</code>, you'll never actually see stdout since input/output of the application are not attached to the console.</p>
<p>Once I changed <code>--deploy-mode</code> to <code>client</code>, I also needed to add <code>--conf spark.driver.host</code> as documented <a href="https://github.com/bitnami/charts/issues/2883" rel="nofollow noreferrer">here</a> and <a href="https://github.com/bitnami/charts/issues/2946" rel="nofollow noreferrer">here</a>, for the pods to be able to resolve back to the invoking host.</p>
<pre class="lang-sh prettyprint-override"><code>kubectl run -n redacted spark-client --rm -it --restart='Never' \
  --image docker.io/bitnami/spark:3.2.0-debian-10-r2 \
  -- /bin/bash -c '
run-example \
  --name my-pi-calc-example \
  --master spark://spark-master-svc:7077 \
  --deploy-mode client \
  --conf spark.driver.host=$(hostname -i) \
  SparkPi 10'
</code></pre>
<p>Output:</p>
<pre><code>21/11/15 23:22:16 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
21/11/15 23:22:16 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 2.961188 s
Pi is roughly 3.140959140959141
21/11/15 23:22:16 INFO SparkUI: Stopped Spark web UI at http://xx.xx.xx.xx:4040
21/11/15 23:22:16 INFO StandaloneSchedulerBackend: Shutting down all executors
21/11/15 23:22:16 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
</code></pre>
<p>Interestingly enough this is <code>app-20211115232213-0024</code>, which in the Spark Master UI still shows each worker as KILLED 143 - reinforcing the conclusion that that is a 'normal' shutdown signal.</p>
