<p>I have Spark Jobs running on Yarn. These days I'm moving to Spark on Kubernetes.</p>
<p>On Kubernetes I'm having an issue: files uploaded via <code>--files</code> can't be read by Spark Driver.</p>
<p>On Yarn, as described in many answers I can read those files using <code>Source.fromFile(filename)</code>.</p>
<p>But I can't read files in Spark on Kubernetes.</p>
<ul>
<li>Spark version: 3.0.1</li>
<li>Scala version: 2.12.6</li>
<li>deploy-mode: cluster</li>
<li>submit commands
<pre class="lang-none prettyprint-override"><code>$ spark-submit --class &lt;className&gt; \
      --name=&lt;jobName&gt; \
      --master=k8s://https://api-hostname:6443 \
      ...
      --deploy-mode=cluster \
      --files app.conf \
      --conf spark.kubernetes.file.upload.path=hdfs://&lt;nameservice&gt;/path/to/sparkUploads/ \
      app.jar
</code></pre>
</li>
</ul>
<p>After executing above command, <code>app.conf</code> is uploaded to <code>hdfs://&lt;nameservice&gt;/path/to/sparkUploads/spark-upload-xxxxxxx/</code>,</p>
<p>And in Driver's pod, I found <code>app.conf</code> in <code>/tmp/spark-******/</code> directory, <code>app.jar</code> as well.</p>
<p>But Driver can't read <code>app.conf</code>, <code>Source.fromFile(filename)</code> returns <code>null</code>, there was no permission problems.</p>
<h3>Update 1</h3>
<p>In Spark Web UI-&gt;&quot;Environment&quot; Tab, <code>spark://&lt;pod-name&gt;-svc.ni.svc:7078/files/app.conf</code> in &quot;Classpath Entries&quot; menu. Does this mean <code>app.conf</code> is available in classpath?</p>
<p>On the other hand, in Spark on Yarn <code>user.dir</code> property was included in System classpath.</p>
<hr />
<p>I found <a href="https://issues.apache.org/jira/browse/SPARK-31726" rel="nofollow noreferrer">SPARK-31726: Make spark.files available in driver with cluster deploy mode on kubernetes</a></p>
<h3>Update 2</h3>
<p>I found that driver pod's <code>/opt/spark/work-dir/</code> dir was included in classpath.</p>
<p>but <code>/opt/spark/work-dir/</code> is empty on driver pod whereas on executor pod it contains <code>app.conf</code> and <code>app.jar</code>.</p>
<p>I think that is the problem and <a href="https://issues.apache.org/jira/browse/SPARK-31726" rel="nofollow noreferrer">SPARK-31726</a> describes this.</p>
<h3>Update 3</h3>
<p>After reading Jacek's answer, I tested <code>org.apache.spark.SparkFiles.getRootDirectory()</code>.</p>
<p>It returns <code>/var/data/spark-357eb33e-1c17-4ad4-b1e8-6f878b1d8253/spark-e07d7e84-0fa7-410e-b0da-7219c412afa3/userFiles-59084588-f7f6-4ba2-a3a3-9997a780af24</code></p>
<h3>Update 4 - work around</h3>
<ul>
<li>First, I make ConfigMaps to save files that I want to read driver/executors</li>
<li>Next, The ConfigMaps are mounted on driver/executors. To mount ConfigMap, use <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template" rel="nofollow noreferrer">Pod Template</a> or <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#mounting-configmaps" rel="nofollow noreferrer">Spark Operator</a></li>
</ul>

Jason Heo
<p><code>--files</code> files should be accessed using <a href="http://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkFiles$.html#get(filename:String):String" rel="nofollow noreferrer">SparkFiles.get</a> utility:</p>
<blockquote>
<p><strong>get(filename: String): String</strong></p>
<p>Get the absolute path of a file added through <code>SparkContext.addFile()</code>.</p>
</blockquote>
