<p>I'm trying to get off the ground with Spark and Kubernetes but I'm facing difficulties. I used the helm chart here:</p>
<p><a href="https://github.com/bitnami/charts/tree/main/bitnami/spark" rel="nofollow noreferrer">https://github.com/bitnami/charts/tree/main/bitnami/spark</a></p>
<p>I have 3 workers and they all report running successfully. I'm trying to run the following program remotely:</p>
<pre><code>from pyspark.sql import SparkSession
spark = SparkSession.builder.master(&quot;spark://&lt;master-ip&gt;:&lt;master-port&gt;&quot;).getOrCreate()
df = spark.read.json('people.json')
</code></pre>
<p>Here's the part that's not entirely clear. Where should the file people.json actually live? I have it locally where I'm running the python code and I also have it on a PVC that the master and all workers can see at /sparkdata/people.json.</p>
<p>When I run the 3rd line as simply <code>'people.json'</code> then it starts running but errors out with:</p>
<p><code>WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources</code></p>
<p>If I run it as <code>'/sparkdata/people.json'</code> then I get</p>
<p><code>pyspark.sql.utils.AnalysisException: Path does not exist: file:/sparkdata/people.json</code></p>
<p>Not sure where I go from here. To be clear I want it to read files from the PVC. It's an NFS share that has the data files on it.</p>

<p>Your <code>people.json</code> file needs to be accessible to your driver + executor pods. This can be achieved in multiple ways:</p>
<ul>
<li>having some kind of network/cloud drive that each pod can access</li>
<li>mounting volumes on your pods, and then uploading the data to those volumes using <code>--files</code> in your spark-submit.</li>
</ul>
<p>The latter option might be the simpler to set up. <a href="https://jaceklaskowski.github.io/spark-kubernetes-book/demo/spark-and-local-filesystem-in-minikube/#hostpath" rel="nofollow noreferrer">This page</a> discusses in more detail how you could do this, but we can shortly go to the point. If you add the following arguments to your spark-submit you should be able to get your <code>people.json</code> on your driver + executors (you just have to choose sensible values for the $VAR variables in there):</p>
<pre><code>  --files people.json \
  --conf spark.kubernetes.file.upload.path=$SOURCE_DIR \
  --conf spark.kubernetes.driver.volumes.$VOLUME_TYPE.$VOLUME_NAME.mount.path=$MOUNT_PATH \
  --conf spark.kubernetes.driver.volumes.$VOLUME_TYPE.$VOLUME_NAME.options.path=$MOUNT_PATH \
  --conf spark.kubernetes.executor.volumes.$VOLUME_TYPE.$VOLUME_NAME.mount.path=$MOUNT_PATH \
  --conf spark.kubernetes.executor.volumes.$VOLUME_TYPE.$VOLUME_NAME.options.path=$MOUNT_PATH \
</code></pre>
<p>You can always verify the existence of your data by going inside of the pods themselves like so:</p>
<pre><code>kubectl exec -it &lt;driver/executor pod name&gt; bash
(now you should be inside of a bash process in the pod)
cd &lt;mount-path-you-chose&gt;
ls -al
</code></pre>
<p>That last <code>ls -al</code> command should show you a <code>people.json</code> file in there (after having done your spark-submit of course).</p>
<p>Hope this helps!</p>
