title: Logging of pyspark application- where is saved?
body: <p>I have a PySpark job that was submitted to Yarn by airflow using a <code>SparkSubmitOperator</code>. In the python file <code>test.py</code> I have this logging:</p>
  <pre><code>import logging
  logger = logging.getLogger(&quot;myapp&quot;)
  logger.info(&quot;this is to log&quot;)
  </code></pre>
  <p>The opreator looks like this:</p>
  <pre><code>spark_etl= SparkSubmitOperator(
       task_id = &quot;etl_job&quot;,
       name = &quot;transform files&quot;,
       application = &quot;test.py&quot;,
       .... 
  </code></pre>
  <p>I checked the application log in the Yarn application manager, but the log was not printed out there. I checked the log for this airflow task, it was not printed out there either. Could you please help me to understand how/ where is PySpark application log is saved? Many thanks for your help.</p>