title: Execute multiple Spark jobs
body: <p>I am running a Spark job with following cluster and application configuration:</p>
  
  <p>Total Node: 3
  <code>
  Master Node Memory 7.5GB, 2 Cores
  Worker Node1, Memory 15GB, 4 Cores
  Worker Node2, Memory 15GB, 4 Cores</code></p>
  
  <p>Application Configuration:</p>
  
  <pre><code>--master yarn --num-executors 2 --executor-cores 2 --executor-memory 2G
  </code></pre>
  
  <p>I am trying to submit multiple jobs in same time with same user, however I see only first two submitted jobs are executing and third has to wait with following warring.</p>
  
  <pre><code>19/11/19 08:30:49 WARN org.apache.spark.util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
  19/11/19 08:30:49 WARN org.apache.spark.util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
  </code></pre>
  
  <p>I found that SparkUI is being created for every submitted job, and my cluster is accepting only two job a time. Further I observed that it picked up the third job on port 4042 once first submitted job finished the execution. What could be wrong with my cluster that it is accepting only two job at a time?</p>
  
  <p>Here is my Spark Session Code:</p>
  
  <pre><code>val spark: SparkSession = {
    val sparkSession = SparkSession
      .builder()
      .appName("Data Analytics")
      .config("spark.scheduler.mode", "FAIR")
      //.master("local[*]")
      .getOrCreate()
    sparkSession
  }
  </code></pre>
  
  <p>Further my questions are:
  Why SparkSession is creating SparkUI for each job and how we can solve this problem.
  Is there any way to use same Session for multiple jobs.</p>