<p>I have a K8s cluster up and running, on VMs inside VMWare Workstation, as of now. I'm trying to deploy a Spark application natively using the official documentation from <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template" rel="nofollow noreferrer">here</a>. However, I also landed on <a href="https://towardsdatascience.com/how-to-build-spark-from-source-and-deploy-it-to-a-kubernetes-cluster-in-60-minutes-225829b744f9?gi=b30ed2bf7f9a" rel="nofollow noreferrer">this</a> article which made it clearer, I felt.</p>

<p>Now, earlier my setup was running inside nested VMs, basically my machine is on Win10 and I had an Ubuntu VM inside which I had 3 more VMs running for the cluster (not the best idea, I know). </p>

<p>When I tried to run my setup by following the article mentioned, I first created a service account inside the cluster called <code>spark</code>, then created a <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding" rel="nofollow noreferrer">clusterrolebinding</a> called <code>spark-role</code>, gave <code>edit</code> as the clusterrole and assigned it to the <code>spark</code> service account so that Spark driver pod has sufficient permissions.</p>

<p>I then try to run the example SparkPi job using this command line:</p>

<pre><code>bin/spark-submit \
  --master k8s://https://&lt;k8-cluster-ip&gt;:&lt;k8-cluster-port&gt; \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=kmaster:5000/spark:latest \
  --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 100
</code></pre>

<p>And it fails within a few seconds after it has created the <strong>driver-pod</strong>, it goes into <strong>Running</strong> state and after like 3 seconds goes into <strong>Error</strong> state. </p>

<p>On giving the command <code>kubectl logs spark-pi-driver</code> <a href="https://pastebin.com/qs6UKw9k" rel="nofollow noreferrer">this</a> is the log I get.</p>

<p>The second <code>Caused by:</code> is always either as mentioned above i.e:</p>

<ul>
<li><code>Caused by: java.net.SocketException: Broken pipe (Write failed)</code> or,</li>
<li><code>Caused by: okhttp3.internal.http2.ConnectionShutdownException</code></li>
</ul>

<p><a href="https://pastebin.com/spgfM6ie" rel="nofollow noreferrer">Log #2</a> for reference.</p>

<p>After running into dead-ends with this, I tried giving <code>--deploy-mode client</code> to see if it makes a difference and get more verbose logs. You can read the difference between client and cluster mode from <a href="https://techvidvan.com/tutorials/spark-modes-of-deployment/" rel="nofollow noreferrer">here</a>. </p>

<p>On deploying the job as <strong>client mode</strong> it still fails, however, now I see that each time the driver pod (now running not as a pod but as a process on the local machine) tries to create an executor pod, it goes into a loop infinitely trying to create an executor pod with a count-number appended to the pod name, as the last one goes into a terminated state. Also, now I can see the Spark UI on the 4040 port but the job doesn't move forward as it's stuck on trying to create even a single executor pod. </p>

<p>I get <a href="https://pastebin.com/LmbA67GR" rel="nofollow noreferrer">this</a> log.</p>

<p>To me, this makes it pretty apparent that it's a resource crunch maybe?</p>

<p>So to be sure, I delete the nested VMs and setup 2 new VMs on my main machine and connect them using a NAT network and setup the same K8s cluster.</p>

<p>But now when I try to do the exact same thing it fails with the same error (Broken Pipe/ShutdownException), except now it tells me that it fails even at <strong>creating a driver-pod</strong>. </p>

<p><a href="https://pastebin.com/sWS6vBQf" rel="nofollow noreferrer">This</a> is the log for reference.</p>

<p>Now I can't even fetch logs as to why it fails, because it's never even created.</p>

<p>I've broken my head over this and can't figure out why it's failing. Now, I tried out a lot of things to rule them out but so far nothing has worked except one (which is a completely different solution). </p>

<p>I tried the <strong>spark-on-k8-operator</strong> from GCP from <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">here</a> and it <strong>worked</strong> for me. I wasn't able to see the Spark UI as it runs briefly but it prints the Pi value in the shell window, so I know it works.
I'm guessing, that even this spark-on-k8s-operator 'internally' does the same thing but I really need to be able to deploy it natively, or at least know why it fails.</p>

<p>Any help here will be appreciated (I know it's a long post). Thank you.</p>

<p>Make sure the kubernetes version that you are deploying is compatible with the Spark version that you are using.</p>

<p>Apache Spark uses the Kubernetes Client library to communicate with the kubernetes cluster.</p>

<p>As per today the latest LTS Spark version is 2.4.5 which includes the kubernetes client version 4.6.3.</p>

<p>Checking the <strong>compatibility matrix</strong> of Kubernetes Client: <a href="https://github.com/fabric8io/kubernetes-client/blob/master/README.md#compatibility-matrix" rel="nofollow noreferrer">here</a></p>

<p>The supported kubernetes versions go all the way up to v1.17.0.</p>

<p>Based on my personal experience <strong>Apache Spark 2.4.5 works well with kubernetes version v1.15.3</strong>. I have had problems with more recent versions.</p>

<p>When a not supported kubernetes version is used, the logs to get are as the ones you are describing:</p>

<pre><code>Caused by: java.net.SocketException: Broken pipe (Write failed) or,
Caused by: okhttp3.internal.http2.ConnectionShutdownException
</code></pre>
