<p>I am trying to setup a spark cluster on k8s. I've managed to create and setup a cluster with three nodes by following this article: 
<a href="https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/" rel="noreferrer">https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/</a></p>

<p>After that when I tried to deploy spark on the cluster it failed at spark submit setup. 
I used this command:</p>

<pre><code>~/opt/spark/spark-2.3.0-bin-hadoop2.7/bin/spark-submit \
--master k8s://https://206.189.126.172:6443 \
--deploy-mode cluster \
--name word-count \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=docker.io/garfiny/spark:v2.3.0 \
—-conf spark.kubernetes.driver.pod.name=word-count \
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
</code></pre>

<p>And it gives me this error:</p>

<pre><code>Exception in thread "main" org.apache.spark.SparkException: The Kubernetes mode does not yet support referencing application dependencies in the local file system.
    at org.apache.spark.deploy.k8s.submit.DriverConfigOrchestrator.getAllConfigurationSteps(DriverConfigOrchestrator.scala:122)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:229)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:227)
    at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2585)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:227)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:192)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
</code></pre>

<p>2018-06-04 10:58:24 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-06-04 10:58:24 INFO  ShutdownHookManager:54 - Deleting directory /private/var/folders/lz/0bb8xlyd247cwc3kvh6pmrz00000gn/T/spark-3967f4ae-e8b3-428d-ba22-580fc9c840cd</p>

<p>Note: I followed this article for installing spark on k8s.
<a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html</a></p>

garfiny
<p>The error message comes from <a href="https://github.com/apache/spark/commit/5d7c4ba4d73a72f26d591108db3c20b4a6c84f3f" rel="noreferrer">commit 5d7c4ba4d73a72f26d591108db3c20b4a6c84f3f</a> and include the page you mention: "<a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management" rel="noreferrer">Running Spark on Kubernetes</a>" with the mention that you indicate:</p>



<pre class="lang-scala prettyprint-override"><code>// TODO(SPARK-23153): remove once submission client local dependencies are supported.
if (existSubmissionLocalFiles(sparkJars) || existSubmissionLocalFiles(sparkFiles)) {
  throw new SparkException("The Kubernetes mode does not yet support referencing application " +
    "dependencies in the local file system.")
}
</code></pre>

<p>This is described in <a href="https://issues.apache.org/jira/browse/SPARK-18278" rel="noreferrer">SPARK-18278</a>:</p>

<blockquote>
  <p>it wouldn't accept running a local: jar file, e.g. <code>local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar</code>, on my spark docker image (<code>allowsMixedArguments</code> and <code>isAppResourceReq booleans</code> in <code>SparkSubmitCommandBuilder.java</code> get in the way). </p>
</blockquote>

<p>And this is linked to <a href="https://github.com/kubernetes/kubernetes/issues/34377" rel="noreferrer">kubernetes issue 34377</a></p>

<p>The <a href="https://issues.apache.org/jira/browse/SPARK-22962" rel="noreferrer">issue SPARK-22962 "Kubernetes app fails if local files are used"</a> mentions:</p>

<blockquote>
  <p>This is the resource staging server use-case. We'll upstream this in the 2.4.0 timeframe.</p>
</blockquote>

<p>In the meantime, that error message was introduced in <a href="https://github.com/apache/spark/pull/20320" rel="noreferrer">PR 20320</a>.</p>

<p>It includes the comment:</p>

<blockquote>
  <p>The manual tests I did actually use a main app jar located on gcs and http.<br>
  To be specific and for record, I did the following tests:</p>
  
  <ul>
  <li>Using a gs:// main application jar and a http:// dependency jar. Succeeded.</li>
  <li>Using a https:// main application jar and a http:// dependency jar. Succeeded.</li>
  <li>Using a local:// main application jar. Succeeded.</li>
  <li>Using a file:// main application jar. Failed.</li>
  <li>Using a file:// dependency jar. Failed.</li>
  </ul>
</blockquote>

<p>That issue should been fixed by now, and the <a href="https://stackoverflow.com/users/714376/garfiny">OP garfiny</a> confirms <a href="https://stackoverflow.com/questions/50637250/spark-on-k8s-getting-error-kube-mode-not-support-referencing-app-depenpendcie/50673683?noredirect=1#comment88396515_50673683">in the comments</a>:</p>

<blockquote>
  <p>I used the newest <code>spark-kubernetes jar</code> to replace the one in <code>spark-2.3.0-bin-hadoop2.7</code> package. The exception is gone.</p>
</blockquote>
