title: How do I log from my Python Spark script
body: <p>I have a Python Spark program which I run with <code>spark-submit</code>. I want to put logging statements in it.</p>
  
  <pre><code>logging.info("This is an informative message.")
  logging.debug("This is a debug message.")
  </code></pre>
  
  <p>I want to use the same logger that Spark is using so that the log messages come out in the same format and the level is controlled by the same configuration files. How do I do this?</p>
  
  <p>I've tried putting the <code>logging</code> statements in the code and starting out with a <code>logging.getLogger()</code>. In both cases I see Spark's log messages but not mine. I've been looking at the <a href="https://docs.python.org/2/library/logging.html">Python logging documentation</a>, but haven't been able to figure it out from there.</p>
  
  <p>Not sure if this is something specific to scripts submitted to Spark or just me not understanding how logging works.</p>