title: Logging from pyspark application to a local or hdfs file
body: <p>I have an application in pyspark includes closure functions that contain logging statements, I don't know how to log messages to local/hdfs file in pyspark.
  I tried something as below but doesn't work: </p>
  
  <pre><code>import json
  from pyspark import SparkContext
  import logging
  
  def parse_json(text_line):
      try:
          return(json.loads(text_line))
      except ValueError:
          # here I need to log a warning message to a local file or even to default spark logs
          logger.warn("invalid json structure" + text_line)
          return({})
  
  if __name__ == "__main__":
      my_data = ['{"id": "111", "name": "aaa"}',
                 '{"wrong json", "name": "bbb"}',
                 '{"id": "333", "name": "ccc"}']
      sc = SparkContext()
      logger = logging.getLogger('py4j')
      lines = sc.parallelize(my_data)
      my_data_json = lines.map(parse_json).filter(lambda x: x)
      print(my_data_json.collect())
  </code></pre>
  
  <p>Any help please!</p>