title: How do I create SparkSession from SparkContext in PySpark?
body: <p>I have a SparkContext <code>sc</code> with a highly customised SparkConf(). How do I use that SparkContext to create a SparkSession? I found this post: <a href="https://stackoverflow.com/a/53633430/201657">https://stackoverflow.com/a/53633430/201657</a> that shows how to do it using Scala:</p>
  
  <pre><code>val spark = SparkSession.builder.config(sc.getConf).getOrCreate()
  </code></pre>
  
  <p>but when I try and apply the same technique using PySpark:</p>
  
  <pre><code>from pyspark.sql import SparkSession
  spark = SparkSession.builder.config(sc.getConf()).enableHiveSupport().getOrCreate()
  </code></pre>
  
  <p>It fails with error </p>
  
  <blockquote>
    <p>AttributeError: 'SparkConf' object has no attribute '_get_object_id'  </p>
  </blockquote>
  
  <p><a href="https://i.stack.imgur.com/87DPf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/87DPf.png" alt="enter image description here"></a></p>
  
  <p>As I say I want to use the same <code>SparkConf</code> in my SparkSession as used in the SparkContext. How do I do it?</p>
  
  <hr>
  
  <p>UPDATE</p>
  
  <p>I've done a bit of fiddling about:</p>
  
  <pre><code>from pyspark.sql import SparkSession
  spark = SparkSession.builder.enableHiveSupport().getOrCreate()
  sc.getConf().getAll() == spark.sparkContext.getConf().getAll()
  </code></pre>
  
  <p>returns</p>
  
  <blockquote>
    <p>True  </p>
  </blockquote>
  
  <p>so the SparkConf of both the SparkContext &amp; the SparkSession are the same. My assumption from this is that <code>SparkSession.builder.getOrCreate()</code> will use an existing SparkContext if it exists. Am I correct?</p>