<p>I am using spark 2.4.4. Having trouble running spark-submit on my ubuntu machine with the following config and system environment:</p>

<pre><code>cpchung:small$ minikube start --kubernetes-version=1.17.0
😄  minikube v1.6.2 on Ubuntu 19.10
✨  Selecting 'kvm2' driver from user configuration (alternates: [virtualbox none])
💡  Tip: Use 'minikube start -p &lt;name&gt;' to create a new cluster, or 'minikube delete' to delete this one.
🏃  Using the running kvm2 "minikube" VM ...
⌛  Waiting for the host to be provisioned ...
🐳  Preparing Kubernetes v1.17.0 on Docker '18.09.9' ...
💾  Downloading kubelet v1.17.0
💾  Downloading kubeadm v1.17.0
🚜  Pulling images ...
🚀  Launching Kubernetes ... 
🏄  Done! kubectl is now configured to use "minikube"
cpchung:small$ minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.39.246:2376"
export DOCKER_CERT_PATH="/home/cpchung/.minikube/certs"
# Run this command to configure your shell:
# eval $(minikube docker-env)
cpchung:small$ eval $(minikube docker-env)
cpchung:small$ ./run.sh 
</code></pre>

<p>inside my <code>run.sh</code> is this:</p>

<pre><code>spark-submit --class SimpleApp \
--master "k8s://https://192.168.39.246:8443" \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=somename:latest \
--conf spark.kubernetes.image.pullPolicy=IfNotPresent \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///small.jar

</code></pre>

<p>Then I got this:</p>

<pre><code>simpleapp-1580791262398-driver   0/1     Error    0          21s
cpchung:small$ kubectl logs simpleapp-1580791262398-driver

20/02/04 04:41:10 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/02/04 04:41:10 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://simpleapp-1580791262398-driver-svc.default.svc:4040
20/02/04 04:41:10 INFO SparkContext: Added JAR file:///small.jar at spark://simpleapp-1580791262398-driver-svc.default.svc:7078/jars/small.jar with timestamp 1580791270603
20/02/04 04:41:14 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes.
20/02/04 04:41:14 WARN WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 - 
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
</code></pre>

cpchung
<p>Spark up to <code>v2.4.4</code> uses <a href="https://github.com/fabric8io/kubernetes-client" rel="nofollow noreferrer">fabric8 Kubernetes client</a> <code>v4.1.2</code> which works only with Kubernetes APIs <code>1.9.0 - 1.12.0</code>, please refer <a href="https://github.com/fabric8io/kubernetes-client#compatibility-matrix" rel="nofollow noreferrer">compatibility matrix</a> and <a href="https://github.com/apache/spark/blob/v2.4.4/resource-managers/kubernetes/core/pom.xml#L32" rel="nofollow noreferrer">pom.xml</a>.</p>
<p>Spark 2.4.5 will upgrade Kubernetes client to <code>v4.6.1</code> and will support Kubernetes APIs up to <code>1.15.2</code>.</p>
<p>So you have the following options:</p>
<ul>
<li>Downgrade Kubernetes cluster to <code>1.12</code></li>
<li>Wait until Spark <code>2.4.5/3.0.0</code> is released and downgrade Kubernetes cluster to <code>1.15.2</code></li>
<li>Upgrade Spark <code>fabric8 Kubernetes client</code> dependency and do custom build of Spark and its Docker image</li>
</ul>
<p>Hope it helps.</p>
<h2>Update 1</h2>
<p>In order to run Spark on Kubernetes cluster with API version <code>1.17.0+</code> you need to patch Spark code based on tag <code>v2.4.5</code> with the following changes:</p>
<pre><code>diff --git a/resource-managers/kubernetes/core/pom.xml b/resource-managers/kubernetes/core/pom.xml
index c9850ca512..41468e1363 100644
--- a/resource-managers/kubernetes/core/pom.xml
+++ b/resource-managers/kubernetes/core/pom.xml
@@ -29,7 +29,7 @@
   &lt;name&gt;Spark Project Kubernetes&lt;/name&gt;
   &lt;properties&gt;
     &lt;sbt.project.name&gt;kubernetes&lt;/sbt.project.name&gt;
-    &lt;kubernetes.client.version&gt;4.6.1&lt;/kubernetes.client.version&gt;
+    &lt;kubernetes.client.version&gt;4.7.1&lt;/kubernetes.client.version&gt;
   &lt;/properties&gt;
 
   &lt;dependencies&gt;
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
index 575bc54ffe..c180ea3a7b 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
@@ -67,7 +67,8 @@ private[spark] class BasicDriverFeatureStep(
       .withAmount(driverCpuCores)
       .build()
     val driverMemoryQuantity = new QuantityBuilder(false)
-      .withAmount(s&quot;${driverMemoryWithOverheadMiB}Mi&quot;)
+      .withAmount(driverMemoryWithOverheadMiB.toString)
+      .withFormat(&quot;Mi&quot;)
       .build()
     val maybeCpuLimitQuantity = driverLimitCores.map { limitCores =&gt;
       (&quot;cpu&quot;, new QuantityBuilder(false).withAmount(limitCores).build())
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
index d89995ba5e..9c589ace92 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
@@ -86,7 +86,8 @@ private[spark] class BasicExecutorFeatureStep(
     // executorId
     val hostname = name.substring(Math.max(0, name.length - 63))
     val executorMemoryQuantity = new QuantityBuilder(false)
-      .withAmount(s&quot;${executorMemoryTotal}Mi&quot;)
+      .withAmount(executorMemoryTotal.toString)
+      .withFormat(&quot;Mi&quot;)
       .build()
     val executorCpuQuantity = new QuantityBuilder(false)
       .withAmount(executorCoresRequest)
</code></pre>
<p><code>QuantityBuilder</code> has changed the way of parsing the input with <a href="https://github.com/fabric8io/kubernetes-client/pull/1921/files" rel="nofollow noreferrer">this PR</a> which is available since fabric8 Kubernetes client <code>v4.7.0</code>.</p>
<h2>Update 2</h2>
<p>Spark images are based on <code>openjdk:8-jdk-slim</code> which runs Java <code>8u252</code> which in order has a <a href="https://github.com/fabric8io/kubernetes-client/issues/2145" rel="nofollow noreferrer">bug related to OkHttp</a>. To fix it we require fabric8 Kubernetes client <code>v4.9.2</code>, please refer its <a href="https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2" rel="nofollow noreferrer">release notes</a> for more details.</p>
<p>Also the patch above can be simplified:</p>
<pre><code>diff --git a/resource-managers/kubernetes/core/pom.xml b/resource-managers/kubernetes/core/pom.xml
index c9850ca512..595aaba8dd 100644
--- a/resource-managers/kubernetes/core/pom.xml
+++ b/resource-managers/kubernetes/core/pom.xml
@@ -29,7 +29,7 @@
   &lt;name&gt;Spark Project Kubernetes&lt;/name&gt;
   &lt;properties&gt;
     &lt;sbt.project.name&gt;kubernetes&lt;/sbt.project.name&gt;
-    &lt;kubernetes.client.version&gt;4.6.1&lt;/kubernetes.client.version&gt;
+    &lt;kubernetes.client.version&gt;4.9.2&lt;/kubernetes.client.version&gt;
   &lt;/properties&gt;
 
   &lt;dependencies&gt;
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
index 575bc54ffe..a1559e07a4 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
@@ -66,9 +66,7 @@ private[spark] class BasicDriverFeatureStep(
     val driverCpuQuantity = new QuantityBuilder(false)
       .withAmount(driverCpuCores)
       .build()
-    val driverMemoryQuantity = new QuantityBuilder(false)
-      .withAmount(s&quot;${driverMemoryWithOverheadMiB}Mi&quot;)
-      .build()
+    val driverMemoryQuantity = new Quantity(s&quot;${driverMemoryWithOverheadMiB}Mi&quot;)
     val maybeCpuLimitQuantity = driverLimitCores.map { limitCores =&gt;
       (&quot;cpu&quot;, new QuantityBuilder(false).withAmount(limitCores).build())
     }
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
index d89995ba5e..b439ebf837 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
@@ -85,9 +85,7 @@ private[spark] class BasicExecutorFeatureStep(
     // name as the hostname.  This preserves uniqueness since the end of name contains
     // executorId
     val hostname = name.substring(Math.max(0, name.length - 63))
-    val executorMemoryQuantity = new QuantityBuilder(false)
-      .withAmount(s&quot;${executorMemoryTotal}Mi&quot;)
-      .build()
+    val executorMemoryQuantity = new Quantity(s&quot;${executorMemoryTotal}Mi&quot;)
     val executorCpuQuantity = new QuantityBuilder(false)
       .withAmount(executorCoresRequest)
       .build()

</code></pre>
