title: Spark-submit with properties file for python
body: <p>I am trying to load spark configuration through the properties file using the following command in Spark 2.4.0.</p>
  <p><code>spark-submit --properties-file props.conf sample.py</code></p>
  <p>It gives the following error</p>
  <pre><code>org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.
  </code></pre>
  <p>The <code>props.conf</code> file has this</p>
  <pre><code>spark.master    yarn
  spark.submit.deployMode client
  spark.authenticate  true
  spark.sql.crossJoin.enabled true
  spark.dynamicAllocation.enabled true
  spark.driver.memory 4g
  spark.driver.memoryOverhead 2048
  spark.executor.memory   2g
  spark.executor.memoryOverhead   2048
  </code></pre>
  <p>Now, when I try to run the same by adding all arguments to the command itself, it works fine.</p>
  <pre><code>spark2-submit \
  --conf spark.master=yarn \
  --conf spark.submit.deployMode=client \
  --conf spark.authenticate=true \
  --conf spark.sql.crossJoin.enabled=true \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.driver.memory=4g \
  --conf spark.driver.memoryOverhead=2048 \
  --conf spark.executor.memory=2g \
  --conf spark.executor.memoryOverhead=2048 \
  sample.py
  </code></pre>
  <p>This works as expected.</p>