title: spark-submit, how to specify log4j.properties
body: <p>In spark-submit, how to specify log4j.properties ?</p>
  
  <p>Here is my script. I have tried all of combinations and even just use one local node. but looks like the log4j.properties is not loaded, all debug level info was dumped.  </p>
  
  <pre><code>current_dir=/tmp
  DRIVER_JAVA_OPTIONS="-Dlog4j.configuration=file://${current_dir}/log4j.properties "
  
  spark-submit \
  --conf "spark.driver.extraClassPath=$current_dir/lib/*"  \
  --conf "spark.driver.extraJavaOptions=-Djava.security.krb5.conf=${current_dir}/config/krb5.conf -Djava.security.auth.login.config=${current_dir}/config/mssqldriver.conf" \
  --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file://${curent_dir}/log4j.properties " \
  --class "my.AppMain" \
  --files ${current_dir}/log4j.properties \
  --master local[1] \
  --driver-java-options "$DRIVER_JAVA_OPTIONS" \
  --num-executors 4 \
  --driver-memory 16g \
  --executor-cores 10 \
  --executor-memory 6g \
  $current_dir/my-app-SNAPSHOT-assembly.jar
  </code></pre>
  
  <p>log4j properties:</p>
  
  <pre><code>log4j.rootCategory=INFO, console
  log4j.appender.console=org.apache.log4j.ConsoleAppender
  log4j.appender.console.layout=org.apache.log4j.PatternLayout
  log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
  
  log4j.additivity.org=false
  
  log4j.logger.org=WARN
  parquet.hadoop=WARN
  log4j.logger.com.barcap.eddi=WARN
  log4j.logger.com.barcap.mercury=WARN
  log4j.logger.yarn=WARN
  log4j.logger.io.netty=WARN
  log4j.logger.Remoting=WARN   
  log4j.logger.org.apache.hadoop=ERROR
  
  # this disables the table creation logging which is so verbose
  log4j.logger.hive.ql.parse.ParseDriver=WARN
  
  # this disables pagination nonsense when running in combined mode
  log4j.logger.com.barcap.risk.webservice.servlet.PaginationFactory=WARN
  </code></pre>