<p>Working on a POC for Spark over K8s (Spark ver 2.4.4).</p>

<p>I was able to invoke a spark job using spark-submit:</p>

<pre><code>bin/spark-submit --master k8s://https://localhost:8443 --deploy-mode cluster --conf spark.executor.instances=5 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=liranbo/k8s-test4:latest --class org.apache.spark.examples.SparkPi --name spark-pi local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar 40
</code></pre>

<p>Trying to do the same programmatically using SparkSession builder.</p>

<pre><code>SparkSession.builder().appName(appName).master("k8s://https://localhost:8443")
</code></pre>

<p>And I'm getting the following exception</p>

<pre><code>org.apache.spark.SparkException: Could not parse Master URL: 'k8s://https://localhost:8443'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:493)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at SparkPi.startBySparkSessionBuilder(SparkPi.java:46)
at Runner.main(Runner.java:9)
</code></pre>

<p>Is it possible to set the master url to K8s when using SparkSession? if not why is supported by spark-submit?</p>

<p>BTW: I was able to start the app programmatically using <a href="https://spark.apache.org/docs/2.4.4/api/java/org/apache/spark/launcher/SparkLauncher.html" rel="nofollow noreferrer">SparkLauncher</a>, but this is not the solution I'm after.</p>

<p>I saw a (possibly) working example in <a href="https://stackoverflow.com/questions/50862204/pyspark-sparksession-builder-with-kubernetes-master">sparkPy</a>.</p>

LiranBo
<p>just in case someone runs into the same issue - the issue is related to a missing jar. add the following:</p>

<pre><code>    &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-kubernetes_${scala.version}&lt;/artifactId&gt;
        &lt;version&gt;${spark.version}&lt;/version&gt;
    &lt;/dependency&gt;
</code></pre>
