title: Driver not found exception for org.postgresql.Driver during spark-submit
body: <p>I am trying to learn a Scala-Spark JDBC program on IntelliJ IDEA. In order to do that, I have created a Scala SBT Project and the project structure looks like:</p>
  
  <p>Before writing the JDBC connection parameters in the class, I am trying to load a properties file which contains all my connection properties and trying to display if they are loading properly as below:</p>
  
  <p>testconnection.properties:</p>
  
  <pre><code>devUserName=username
  devPassword=password
  gpDriverClass=org.postgresql.Driver
  gpDevUrl=jdbc:url
  </code></pre>
  
  <p>Code:</p>
  
  <pre><code>package com.yearpartition.obj
  
  import java.io.FileInputStream
  import java.util.Properties
  import org.apache.spark.sql.SparkSession
  import org.apache.spark.SparkConf
  
  object PartitionRetrieval {
  
    var conf = new SparkConf().setAppName("Spark-JDBC")
    val conFile = "/home/hmusr/ReconTest/inputdir/testconnection.properties"
    val properties = new Properties()
    properties.load(new FileInputStream(conFile))
    val connectionUrl = properties.getProperty("gpDevUrl")
    val devUserName=properties.getProperty("devUserName")
    val devPassword=properties.getProperty("devPassword")
    val gpDriverClass=properties.getProperty("gpDriverClass")
  
    println("connectionUrl: " + connectionUrl)
  
    Class.forName(gpDriverClass).newInstance()
  
    def main(args: Array[String]): Unit = {
      val spark = SparkSession.builder().enableHiveSupport().config(conf).master("local[2]").getOrCreate()
      println("connectionUrl: " + connectionUrl)
    }
  }
  </code></pre>
  
  <p>Contents of build.sbt:</p>
  
  <pre><code>libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.0.0" % "provided",
    "org.apache.spark" %% "spark-sql" % "2.0.0" % "provided",
    "org.json4s" %% "json4s-jackson" % "3.2.11" % "provided",
    "org.postgresql" % "postgresql" % "42.1.1" % "provided",
    "org.apache.httpcomponents" % "httpclient" % "4.5.3"
  )
  </code></pre>
  
  <p>When I do the spark-submit I get the exception: </p>
  
  <pre><code>Caused by: java.lang.ClassNotFoundException: org.postgresql.Driver
  </code></pre>
  
  <p>Spark submit command:</p>
  
  <pre><code>SPARK_MAJOR_VERSION=2 spark-submit --class com.yearpartition.obj.PartitionRetrieval yearpartition_2.11-0.1.jar --driver-class-path /home/hmusr/jars/postgresql-42.1.4.jar --jars /home/hmusr/jars/postgresql-42.1.4.jar
  </code></pre>
  
  <p>I have the postgres jar file in the dir: /home/hmusr/jars/ and also in the sbt dependencies.
  Could anyone let me know what is causing this problem and how can I fix it?</p>