title: How does Spark running on YARN account for Python memory usage?
body: <p>After reading through the documentation I do not understand how does Spark running on YARN account for Python memory consumption. </p>
  
  <p>Does it count towards <code>spark.executor.memory</code>, <code>spark.executor.memoryOverhead</code> or where?</p>
  
  <p>In particular I have a PySpark application with <code>spark.executor.memory=25G</code>, <code>spark.executor.cores=4</code> and I encounter frequent <strong>Container killed by YARN for exceeding memory limits.</strong> errors when running a <code>map</code> on an RDD. It operates on a fairly large amount of complex Python objects so it is expected to take up some non-trivial amount of memory but not 25GB. How should I configure the different memory variables for use with heavy Python code?</p>