title: How many SparkSessions can a single application have?
body: <p>I have found that as Spark runs, and tables grow in size (through Joins) that the spark executors will eventually run out of memory and the entire system crashes. Even if I try to write temporary results to Hive tables (on HDFS), the system still doesn't free much memory, and my entire system crashes after about 130 joins.</p>
  
  <p>However, through experimentation, I realized that if I break the problem into smaller pieces, write temporary results to hive tables, and Stop/Start the Spark session (and spark context), then the system's resources are freed. I was able to join over 1,000 columns using this approach.</p>
  
  <p>But I can't find any documentation to understand if this is considered a good practice or not (I know you should not acquire multiple sessions at once). Most systems acquire the session in the beginning and close it in the end. I could also break the application into smaller ones, and use a driver like Oozie to schedule these smaller applications on Yarn. But this approach would start and stop the JVM at each stage, which seems a bit heavy-weight. </p>
  
  <p>So my question: is it bad practice to continually start/stop the spark session to free system resources during the run of a single spark application?</p>
  
  <hr>
  
  <p>But can you elaborate on what you mean by a single SparkContext on a single JVM? I was able call <code>sparkSession.sparkContext().stop()</code>, and also <code>stop</code> the <code>SparkSession</code>. I then created a new <code>SparkSession</code> and used a new <code>sparkContext</code>. No error was thrown.</p>
  
  <p>I was also able to use this on the <code>JavaSparkPi</code> without any problems.</p>
  
  <p>I have tested this in <code>yarn-client</code> and a <code>local</code> spark install.</p>
  
  <p>What exactly does stopping the spark context do, and why can you not create a new one once you've stopped one?</p>