<p>I have the strange problem that a Spark job ran on Kubernetes fails with a lot of "Missing an output location for shuffle X" in jobs where there is a lot of shuffling going on. Increasing executor memory does not help. The same job run on just a single node of the Kubernetes cluster in local[*] mode runs fine however so I suspect it has to do with Kubernetes or underlying Docker.
When an executor dies, the pods are deleted immediately so I cannot track down why it failed. Is there an option that keeps failed pods around so I can view their logs?</p>

rabejens
<p><strong>If you're using the spark executor</strong>:
There is a <code>deleteOnTermination</code> setting in the spark application yaml. See <a href="https://github.com/mesosphere/spark-on-k8s-operator/blob/master/docs/api-docs.md#sparkoperator.k8s.io/v1beta2.ExecutorSpec" rel="nofollow noreferrer">the spark-on-kubernetes README.md</a>.</p>
<blockquote>
<p><code>deleteOnTermination</code> - <em>(Optional)</em>
<code>DeleteOnTermination</code> specify whether executor pods should be deleted in case of failure or normal termination.<br><br> Maps to <code>spark.kubernetes.executor.deleteOnTermination</code> that is available since Spark 3.0.</p>
</blockquote>
<p><strong>If you're using kubernetes jobs</strong>: Set the job <code>spec.ttlSecondsAfterFinished</code> parameter or get the previous pods logs with kubectl. There is also a setting for keeping failed jobs around if you're using cronjobs.</p>
