title: How to set `spark.driver.memory` in client mode - pyspark (version 2.3.1)
body: <p>I'm new to PySpark and I'm trying to use <em>pySpark</em> (ver 2.3.1) on my local computer with <em>Jupyter-Notebook</em>.</p>
  
  <p>I want to set <em>spark.driver.memory</em> to 9Gb by doing this:</p>
  
  <pre><code>spark = SparkSession.builder \
         .master("local[2]") \
         .appName("test") \
         .config("spark.driver.memory", "9g")\
         .getOrCreate()
  sc = spark.sparkContext
  from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  
  spark.sparkContext._conf.getAll()  # check the config
  </code></pre>
  
  <p>It returns</p>
  
  <pre><code>[('spark.driver.memory', '9g'),
  ('spark.driver.cores', '4'),
  ('spark.rdd.compress', 'True'),
  ('spark.driver.port', '15611'),
  ('spark.serializer.objectStreamReset', '100'),
  ('spark.app.name', 'test'),
  ('spark.executor.id', 'driver'),
  ('spark.submit.deployMode', 'client'),
  ('spark.ui.showConsoleProgress', 'true'),
  ('spark.master', 'local[2]'),
  ('spark.app.id', 'local-xyz'),
  ('spark.driver.host', '0.0.0.0')]
  </code></pre>
  
  <p>It's quite of weird because when I look at the document, it shows that </p>
  
  <blockquote>
    <p>Note: In <strong>client mode</strong>, this config <strong>must not be set</strong> through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file. <a href="https://spark.apache.org/docs/2.3.1/configuration.html" rel="noreferrer">document here</a></p>
  </blockquote>
  
  <p>But, as you see in the result above, it returns </p>
  
  <blockquote>
    <p>[('spark.driver.memory', '9g')</p>
  </blockquote>
  
  <p>Even when I access to the <strong>spark web UI</strong> (on port <em>4040</em>, <em>environment</em> tab), it still shows 
  <a href="https://i.stack.imgur.com/EbSbp.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EbSbp.png" alt="enter image description here"></a></p>
  
  <p>I tried one more time, with <code>'spark.driver.memory', '10g'</code>. The <em>web UI</em> and <code>spark.sparkContext._conf.getAll()</code> returned <strong>'10g'</strong>.
  I'm so confused about that.
  My questions are:</p>
  
  <ol>
  <li><p>Is the document right about <code>spark.driver.memory</code> config</p></li>
  <li><p>If the document is right, is there a proper way that I can check <code>spark.driver.memory</code> after config. I tried <code>spark.sparkContext._conf.getAll()</code> as well as <em>Spark web UI</em> but it seems to lead to a wrong answer.</p></li>
  </ol>