title: Apache spark: setting spark.eventLog.enabled and spark.eventLog.dir at submit or Spark start
body: <p>I would like to set <code>spark.eventLog.enabled</code> and <code>spark.eventLog.dir</code> at the <code>spark-submit</code> or <code>start-all</code> level -- not require it to be enabled in the scala/java/python code.
  I have tried various things with no success:</p>
  <h3>Setting <code>spark-defaults.conf</code> as</h3>
  <pre><code>spark.eventLog.enabled           true
  spark.eventLog.dir               hdfs://namenode:8021/directory
  </code></pre>
  <p>or</p>
  <pre><code>spark.eventLog.enabled           true
  spark.eventLog.dir               file:///some/where
  </code></pre>
  <h3>Running <code>spark-submit</code> as:</h3>
  <pre><code>spark-submit --conf &quot;spark.eventLog.enabled=true&quot; --conf &quot;spark.eventLog.dir=file:///tmp/test&quot; --master spark://server:7077 examples/src/main/python/pi.py
  </code></pre>
  <h3>Starting spark with environment variables:</h3>
  <pre><code>SPARK_DAEMON_JAVA_OPTS=&quot;-Dspark.eventLog.enabled=true -Dspark.history.fs.logDirectory=$sparkHistoryDir -Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=2d&quot;
  </code></pre>
  <p>and just for overkill:</p>
  <pre><code>SPARK_HISTORY_OPTS=&quot;-Dspark.eventLog.enabled=true -Dspark.history.fs.logDirectory=$sparkHistoryDir -Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=2d&quot;
  </code></pre>
  <p>Where and how must these things be set to get history on arbitrary jobs?</p>