title: cant override Typesafe configuration on commmandline in spark
body: <p>I have a typesafe configuration <code>application.conf</code> in the <code>src/main/resources</code>folder which is loaded by default.</p>
  
  <p>A single value can be overridden by specifying:</p>
  
  <pre><code>--conf spark.driver.extraJavaOptions=-DsomeValue="foo"
  </code></pre>
  
  <p>However, specifying a complete new, i.e. overriding <code>application.conf</code> file like:</p>
  
  <pre><code>spark-submit \
      --class my.Class \
      --master "local[2]" \
      --files foo.conf \
      --conf spark.driver.extraClassPath="-Dconfig.file=file:foo.conf" \
      --conf spark.driver.extraJavaOptions=-Dvalue="abcd" \
      job.jar
  </code></pre>
  
  <p>will fail to load <code>foo.conf</code>. Instead, the original file from the resources folder will be loaded.
  Trying the tricks from: <a href="https://stackoverflow.com/questions/40507436/using-typesafe-config-with-spark-on-yarn">Using typesafe config with Spark on Yarn</a> did not help as well.</p>
  
  <h1>edit</h1>
  
  <p><a href="https://stackoverflow.com/questions/24966621/overriding-multiple-config-values-in-typesafe-config-when-using-an-uberjar-to-de">Overriding multiple config values in Typesafe config when using an uberjar to deploy</a> seems to be the answer for plain (without spark) programs.
  The question remains how to bring this to spark.</p>
  
  <p>Also passing:</p>
  
  <pre><code>--conf spark.driver.extraClassPath="-Dconfig.resource=file:foo.conf"
  --conf spark.driver.extraClassPath="-Dconfig.resource=foo.conf"
  </code></pre>
  
  <p>fails to load my configuration from the command line .</p>
  
  <p>Though, according to the docs:</p>
  
  <blockquote>
    <p><a href="https://github.com/lightbend/config" rel="nofollow noreferrer">https://github.com/lightbend/config</a> For applications using
    application.{conf,json,properties}, system properties can be used to
    force a different config source (e.g. from command line
    -Dconfig.file=path/to/config-file):</p>
    
    <ul>
    <li>config.resource specifies a resource name - not a basename, i.e. application.conf not application</li>
    <li>config.file specifies a filesystem path, again it should include the extension, not be a basename</li>
    <li>config.url specifies a URL</li>
    </ul>
    
    <p>These system properties specify a replacement for
    application.{conf,json,properties}, not an addition. They only affect
    apps using the default ConfigFactory.load() configuration. In the
    replacement config file, you can use include "application" to include
    the original default config file; after the include statement you
    could go on to override certain settings.</p>
  </blockquote>
  
  <p>it should be possible with these parameters.</p>