title: Best practice to create SparkSession object in Scala to use both in unittest and spark-submit
body: <p>I have tried to write a transform method from DataFrame to DataFrame.
  And I also want to test it by scalatest.</p>
  
  <p>As you know, in Spark 2.x with Scala API, you can create SparkSession object as follows:</p>
  
  <pre><code>import org.apache.spark.sql.SparkSession
  
  val spark = SparkSession.bulider
       .config("spark.master", "local[2]")
       .getOrCreate()
  </code></pre>
  
  <p>This code works fine with unit tests.
  But, when I run this code with spark-submit, the cluster options did not work.
  For example,</p>
  
  <pre><code>spark-submit --master yarn --deploy-mode client --num-executors 10 ...
  </code></pre>
  
  <p>does not create any executors.</p>
  
  <p>I have found that the spark-submit arguments are applied when I remove <code>config("master", "local[2]")</code> part of the above code.
  But, without master setting the unit test code did not work.</p>
  
  <p>I tried to split spark (SparkSession) object generation part to test and main.
  But there is so many code blocks needs spark, for example <code>import spark.implicit,_</code> and <code>spark.createDataFrame(rdd, schema)</code>.</p>
  
  <p>Is there any best practice to write a code to create spark object both to test and to run spark-submit?</p>