title: Increase memory available to PySpark at runtime
body: <p>I'm trying to build a recommender using Spark and just ran out of memory:</p>
  
  <pre><code>Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError: Java heap space
  </code></pre>
  
  <p>I'd like to increase the memory available to Spark by modifying the <code>spark.executor.memory</code> property, in PySpark, at runtime.</p>
  
  <p>Is that possible? If so, how?</p>
  
  <p><strong>update</strong></p>
  
  <p>inspired by the link in @zero323's comment, I tried to delete and recreate the context in PySpark:</p>
  
  <pre><code>del sc
  from pyspark import SparkConf, SparkContext
  conf = (SparkConf().setMaster("http://hadoop01.woolford.io:7077").setAppName("recommender").set("spark.executor.memory", "2g"))
  sc = SparkContext(conf = conf)
  </code></pre>
  
  <p>returned:</p>
  
  <pre><code>ValueError: Cannot run multiple SparkContexts at once;
  </code></pre>
  
  <p>That's weird, since:</p>
  
  <pre><code>&gt;&gt;&gt; sc
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  NameError: name 'sc' is not defined
  </code></pre>