title: Run spark application on a different version of spark remotely
body: <p>I have few spark tests that I am running fine remotely through maven on spark 1.6.0 and am using scala. Now I want to run these tests on spark2. The problem is cloudera which by default is using spark 1.6. <strong>Where is cloudera taking this version from and what do I need to do to change the default version of spark ?</strong> Also, spark 1.6 and spark 2 are present on same cluster. Both spark versions are present on top of yarn. The hadoop config files are present on the cluster which I am using to run the tests on the test environment and This is how I am getting spark context.</p>
  
  <pre><code>def getSparkContext(hadoopConfiguration: Configuration): SparkContext ={
  val conf = new SparkConf().setAppName("SparkTest").setMaster("local")     
  hadoopConfiguration.set("hadoop.security.authentication", "Kerberos")
  UserGroupInformation.loginUserFromKeytab("alice", "/etc/security/keytab/alice.keytab")
  val sc=new SparkContext(conf)
  return sc
  }
  </code></pre>
  
  <p>Is there any way I can specify the version in the conf files or cloudera itself ?</p>