title: Redirect Spark console logs into a file
body: <p>As asked, i would like to keep a trace of sparks master logs to keep errors logs when they happend. I know that there are the workers logs on the webUI, but i'm not sure they show the same kind of error than the master.</p>
  
  <p>I find that we have to modify the conf/log4j.properties but my tries doesn't work..</p>
  
  <p>Default configuration + add file :</p>
  
  <pre><code># Set everything to be logged to the console
  log4j.rootCategory=INFO, console, file
  log4j.appender.console=org.apache.log4j.ConsoleAppender
  log4j.appender.console.target=System.err
  log4j.appender.console.layout=org.apache.log4j.PatternLayout
  log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
  
  # Settings to quiet third party logs that are too verbose
  log4j.logger.org.spark-project.jetty=WARN
  log4j.logger.org.spark-
  project.jetty.util.component.AbstractLifeCycle=ERROR
  log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
  log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
  log4j.logger.org.apache.parquet=ERROR
  log4j.logger.parquet=ERROR
  
  # SPARK-9183: Settings to avoid annoying messages when looking up 
  nonexistent UDFs in SparkSQL with Hive support
  log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
  log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
  </code></pre>
  
  <p>Try to setup the file</p>
  
  <pre><code>###Custom log file
  log4j.appender.file=org.apache.log4j.RollingFileAppender
  log4j.appender.file.fileName=/var/data/log/MasterLogs/master.log
  log4j.appender.file.ImmediateFlush=true
  ## Set the append to false, overwrite
  log4j.appender.file.Append=false
  log4j.appender.file.MaxFileSize=100MB
  log4j.appender.file.MaxBackupIndex=10
  ##Define the layout for file appender
  log4j.appender.file.layout=org.apache.log4j.PatternLayout
  log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
  </code></pre>