title: Can I change Spark's executor memory at runtime?
body: <p>Is it possible to change the value of executor memory at runtime in Spark? The reason I want to do it is that for some map tasks I want the yarn scheduler to put each task on a separate node. By increasing the executor memory to near the total memory of a node, I ensure they are distributed on each node. Later on, I want to run several tasks per node, so I would lower the executor memory for them.</p>