<p>I have created simple word count program jar file which is tested and works fine. However, when I am trying to run the same jar file on my <code>Kubernetes</code> cluster it's throwing an error. Below is my <code>spark-submit</code> code along with the error thrown.  </p>

<pre><code>spark-submit --master k8s://https://192.168.99.101:8443 --deploy-mode cluster --name WordCount --class com.sample.WordCount --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=debuggerrr/spark-new:spark-new local:///C:/Users/siddh/OneDrive/Desktop/WordCountSample/target/WordCountSample-0.0.1-SNAPSHOT.jar local:///C:/Users/siddh/OneDrive/Desktop/initialData.txt  
</code></pre>

<p>The last <code>local</code> argument is the data file on which the wordcount program will run and fetch the results.  </p>

<p>Below is my error:  </p>

<pre><code>    status: [ContainerStatus(containerID=null, image=gcr.io/spark-operator/spark:v2.4.5, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=Back-off pulling image "gcr.io/spark-operator/spark:v2.4.5", reason=ImagePullBackOff, additionalProperties={}), additionalProperties={}), additionalProperties={started=false})]
20/02/11 22:48:13 INFO LoggingPodStatusWatcherImpl: State changed, new state:
         pod name: wordcount-1581441237366-driver
         namespace: default
         labels: spark-app-selector -&gt; spark-386c19d289a54e2da1733376821985b1, spark-role -&gt; driver
         pod uid: a9e74d13-cf77-4de0-a16d-a71a21118ef8
         creation time: 2020-02-11T17:13:59Z
         service account name: default
         volumes: spark-local-dir-1, spark-conf-volume, default-token-wbvkb
         node name: minikube
         start time: 2020-02-11T17:13:59Z
         container images: gcr.io/spark-operator/spark:v2.4.5
         phase: Running
         status: [ContainerStatus(containerID=docker://7b46d9483cf22d94c7553455dd06a6a9530b2947a6db71d089cfe9dcce656c26, image=gcr.io/spark-operator/spark:v2.4.5, imageID=docker-pullable://gcr.io/spark-operator/spark@sha256:0d2c7d9d66fb83a0311442f0d2830280dcaba601244d1d8c1704d72f5806cc4c, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=2020-02-11T17:18:11Z, additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={started=true})]
20/02/11 22:48:19 INFO LoggingPodStatusWatcherImpl: State changed, new state:
         pod name: wordcount-1581441237366-driver
         namespace: default
         labels: spark-app-selector -&gt; spark-386c19d289a54e2da1733376821985b1, spark-role -&gt; driver
         pod uid: a9e74d13-cf77-4de0-a16d-a71a21118ef8
         creation time: 2020-02-11T17:13:59Z
         service account name: default
         volumes: spark-local-dir-1, spark-conf-volume, default-token-wbvkb
         node name: minikube
         start time: 2020-02-11T17:13:59Z
         container images: gcr.io/spark-operator/spark:v2.4.5
         phase: Failed
         status: [ContainerStatus(containerID=docker://7b46d9483cf22d94c7553455dd06a6a9530b2947a6db71d089cfe9dcce656c26, image=gcr.io/spark-operator/spark:v2.4.5, imageID=docker-pullable://gcr.io/spark-operator/spark@sha256:0d2c7d9d66fb83a0311442f0d2830280dcaba601244d1d8c1704d72f5806cc4c, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://7b46d9483cf22d94c7553455dd06a6a9530b2947a6db71d089cfe9dcce656c26, exitCode=1, finishedAt=2020-02-11T17:18:18Z, message=null, reason=Error, signal=null, startedAt=2020-02-11T17:18:11Z, additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={started=false})]
20/02/11 22:48:21 INFO LoggingPodStatusWatcherImpl: Container final statuses:


         Container name: spark-kubernetes-driver
         Container image: gcr.io/spark-operator/spark:v2.4.5
         Container state: Terminated
         Exit code: 1
20/02/11 22:48:21 INFO Client: Application WordCount finished.
20/02/11 22:48:23 INFO ShutdownHookManager: Shutdown hook called
20/02/11 22:48:23 INFO ShutdownHookManager: Deleting directory C:\Users\siddh\AppData\Local\Temp\spark-1a3ee936-d430-4f9d-976c-3305617678df
</code></pre>

<p>How do I resolve this error? How can I pass the local file?<br>
<strong>NOTE</strong>: JAR files and data files are present on my desktop and not in the docker image.</p>

whatsinthename
<p>unfortunately passing local files to the job is not yet available for official release of Spark on Kubernetes. There is one solution in a Spark fork requiring to add <a href="https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html#dependency-management" rel="nofollow noreferrer">Resource Staging Server deployment</a> to the cluster, but it is not included in the released builds.</p>

<p>Why it is not so easy to support? Imagine how to configure the network communication between your machine and Spark Pods in Kubernetes: in order to pull your local jars Spark Pod should be able to access you machine (probably you need to run web-server locally and expose its endpoints), and vice-versa in order to push jar from you machine to the Spark Pod your <code>spark-submit</code> script needs to access Spark Pod (which can be done via Kubernetes Ingress and requires several more components to be integrated).</p>

<p>The solution Spark allows is to store your artefacts (jars) in the http-accessible place, including hdfs-compatible storage systems. Please refer the <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management" rel="nofollow noreferrer">official docs</a>.</p>

<p>Hope it helps.</p>
