<p>I have a trivially small Spark application written in Java that I am trying to run in a K8s cluster using <code>spark-submit</code>. I built an image with Spark binaries, my uber-JAR file with all necessary dependencies (in <code>/opt/spark/jars/my.jar</code>), and a config file (in <code>/opt/spark/conf/some.json</code>).</p>
<p>In my code, I start with</p>
<pre class="lang-java prettyprint-override"><code>SparkSession session = SparkSession.builder()
.appName(&quot;myapp&quot;)
.config(&quot;spark.logConf&quot;, &quot;true&quot;)
.getOrCreate();

Path someFilePath = FileSystems.getDefault().getPath(&quot;/opt/spark/conf/some.json&quot;);
String someString = new String(Files.readAllBytes(someFilePath));
</code></pre>
<p>and get this exception at <code>readAllBytes</code> from the Spark driver:</p>
<pre><code>java.nio.file.NoSuchFileException: /opt/spark/conf/some.json
</code></pre>
<p>If I run my Docker image manually I can definitely see the file <code>/opt/spark/conf/some.json</code> as I expect. My Spark job runs as root so file permissions should not be a problem.</p>
<p>I have been assuming that, since the same Docker image, with the file indeed present, will be used to start the driver (and executors, but I don't even get to that point), the file should be available to my application. Is that not so? Why wouldn't it see the file?</p>

mustaccio
<p>You seem to get this exception from one of your worker nodes, not from the container.</p>
<p>Make sure that you've specified all files needed as <code>--files</code> option for <code>spark-submit</code>.</p>
<pre><code>spark-submit --master yarn --deploy-mode cluster --files &lt;local files dependecies&gt; ...
</code></pre>
<p><a href="https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management</a></p>
