title: Spark job in Java: how to access files from 'resources' when run on a cluster
body: <p>I wrote a Spark job in Java. The job is packaged as a shaded jar and executed:</p>
  
  <pre><code>spark-submit my-jar.jar
  </code></pre>
  
  <p>In the code, there are some files (Freemarker templates) that reside in <code>src/main/resources/templates</code>. When run locally, I'm able access the files:</p>
  
  <pre><code>File[] files = new File("src/main/resources/templates/").listFiles();
  </code></pre>
  
  <p>When the job is run on a cluster, a null-pointer exception is returned when the previous line is executed.</p>
  
  <p>If I run <code>jar tf my-jar.jar</code> I can see that the files are packaged in a <code>templates/</code> folder:</p>
  
  <pre><code> [...]
   templates/
   templates/my_template.ftl
   [...]
  </code></pre>
  
  <p>I'm just unable to read them; I suspect that <code>.listFiles()</code> tries to access the local filesystem on the cluster node, and the files aren't there.</p>
  
  <p>I'm curious to know how I should package files to be used within a self-contained Spark job. I'd rather not copy them to HDFS outside of the job because it becomes messy to maintain.</p>