title: How to save/write Spark error message to log file in Python program script?
body: <p>In my Python/Spark program script, I have </p>
  
  <pre><code>logger = logging.getLogger(results.app_name)
  logger.setLevel(logging.DEBUG)
  # Create file handler which logs even debug messages
  fh = logging.FileHandler(outputPath+".log")
  fh.setLevel(logging.DEBUG)
  # Create console handler with a higher log level
  ch = logging.StreamHandler()
  ch.setLevel(logging.INFO)
  
  sc = spark.sparkContext
  sc.setLogLevel("WARN")
  log4jLogger = sc._jvm.org.apache.log4j
  LOGGER = log4jLogger.LogManager.getLogger(__name__)
  </code></pre>
  
  <p>to add logger info (<code>logger.info()</code>) and <em>expected</em> errors under specific circumstances (<code>logger.error()</code>) to console and a custom log file (i.e., <code>outputPath+".log"</code>). However, when I get unexpected errors (e.g., program crashes, error on line whatever, timed out problems, datasets don't match), they only get printed on the console. </p>
  
  <p>How do I also add debug/unexpected errors to the custom log file? </p>
  
  <p>e.g., I got an error message printed on the console, but I like it to also be written in the log file automatically by the script:</p>
  
  <blockquote>
    <p>Traceback (most recent call last):
      File "/Users/lc/Documents/sparkprogram.py", line 747, in 
        phenotypes, thresholds, r2All, pAll=regression(prsDict,pheno_file, pheno_delim, pheno_columns, pheno_no_header, covarColumns=covar_columns, outputName=outputPath, logger=logger)
      File "/Users/lc/Documents/sparkprogram.py", line 298, in regression
        assert samplesize==phenodata.shape[0], "Unequal sample size in pheno file and genotype data"
    AssertionError: Unequal sample size in pheno file and genotype data</p>
  </blockquote>