title: hivethriftserver built into spark3.0.0. is throwing error "org.postgresql.Driver" was not found in the CLASSPATH
body: <p>I am moving from Spark2.3.2 with an external Hive server to Spark3.0.0 with a built in thrift hive server, however I am having trouble getting the thriftsever budled with spark to find the postgresql client libraries to connect to an external metastore.</p>
  <p>In Spark2.3.3 I simply set the <code>$HIVE_HOME/conf/hive-site.xml</code> options for the metastore, added the jars to <code>$HIVE_HOME/lib</code> and everything worked. In Spark3.0.0 I declared the location of the jars in <code>$SPARK_HOME/conf/hive-site.xml</code> like so..</p>
  <pre><code>&lt;property&gt;
    &lt;name&gt;spark.sql.hive.metastore.jars&lt;/name&gt;
    &lt;value&gt;/usr/lib/postgresql&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
  &lt;/property&gt;
  </code></pre>
  <p>but that did not work. I build spark from source, so I also tried adding the postgresql dependencies to the spark <code>pom.xml</code> and thriftserver <code>pom.xml</code> files, so that maven can pull in the libraries during compile. But that didn't work either, so I'm stuck as to what to try next.</p>
  <p>Here is the thriftserver log showing the error....</p>
  <pre><code>20/06/23 03:54:54 INFO SharedState: loading hive config file: file:/opt/spark-3.0.0/conf/hive-site.xml
  20/06/23 03:54:54 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/spark-3.0.0/spark-warehouse').
  20/06/23 03:54:54 INFO SharedState: Warehouse path is 'file:/opt/spark-3.0.0/spark-warehouse'.
  20/06/23 03:54:55 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
  20/06/23 03:54:55 INFO HiveConf: Found configuration file file:/opt/spark-3.0.0/conf/hive-site.xml
  20/06/23 03:54:56 INFO SessionState: Created HDFS directory: /tmp/hive/ubuntu/c3454b03-bce5-4e0e-8a8b-3d7532470b3c
  20/06/23 03:54:56 INFO SessionState: Created local directory: /tmp/ubuntu/c3454b03-bce5-4e0e-8a8b-3d7532470b3c
  20/06/23 03:54:56 INFO SessionState: Created HDFS directory: /tmp/hive/ubuntu/c3454b03-bce5-4e0e-8a8b-3d7532470b3c/_tmp_space.db
  20/06/23 03:54:56 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/opt/spark-3.0.0/spark-warehouse
  20/06/23 03:54:57 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
  20/06/23 03:54:57 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
  20/06/23 03:54:57 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
  20/06/23 03:54:57 INFO ObjectStore: ObjectStore, initialize called
  20/06/23 03:54:57 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
  20/06/23 03:54:57 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
  20/06/23 03:54:57 ERROR Datastore: Exception thrown creating StoreManager. See the nested exception
  Error creating transactional connection factory
  org.datanucleus.exceptions.NucleusException: Error creating transactional connection factory
      at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:214)
  
  ....
  
  Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the &quot;BONECP&quot; plugin to create a ConnectionPool gave an error : The specified datastore driver (&quot;org.postgresql.Driver&quot;) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
          at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
          at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
          at org.datanucleus.store.rdbms.ConnectionFactoryImpl.&lt;init&gt;(ConnectionFactoryImpl.java:82)
          ... 100 more
  Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver (&quot;org.postgresql.Driver&quot;) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
          at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
          at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)
          at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
  </code></pre>
  <p>Here is my <code>$SPARK_HOME/conf/hive-site.xml</code> file</p>
  <pre><code>&lt;configuration&gt;
  
   &lt;property&gt;
    &lt;name&gt;javax.jdo.option.ConnectionURL&lt;/name&gt;
    &lt;value&gt;jdbc:postgresql://xxxxxxx:54321/db?sslmode=require&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
   &lt;/property&gt;
  
  
   &lt;property&gt;
    &lt;name&gt;javax.jdo.option.ConnectionDriverName&lt;/name&gt;
    &lt;value&gt;org.postgresql.Driver&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;javax.jdo.option.ConnectionUserName&lt;/name&gt;
    &lt;value&gt;xxxxx&lt;/value&gt;
    &lt;description&gt;db user&lt;/description&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;javax.jdo.option.ConnectionPassword&lt;/name&gt;
    &lt;value&gt;xxxxx&lt;/value&gt;
    &lt;description&gt;db password&lt;/description&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;hive.execution.engine&lt;/name&gt;
    &lt;value&gt;spark&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;spark.master&lt;/name&gt;
    &lt;value&gt;spark://127.0.1.1:7077&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;spark.eventLog.enabled&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;spark.eventLog.dir&lt;/name&gt;
    &lt;value&gt;/var/log/spark&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;spark.executor.memory&lt;/name&gt;
    &lt;value&gt;2048m&lt;/value&gt;
   &lt;/property&gt;
  
   &lt;property&gt;
    &lt;name&gt;spark.serializer&lt;/name&gt;
    &lt;value&gt;org.apache.spark.serializer.KryoSerializer&lt;/value&gt;
   &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;hive.server2.use.SSL&lt;/name&gt;
    &lt;value&gt;false&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;spark.sql.hive.metastore.jars&lt;/name&gt;
    &lt;value&gt;builtin&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  <p>In the <code>.bashrc</code> I have set the following env variables</p>
  <pre><code>export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_251
  export SPARK_HOME=/opt/spark
  export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
  </code></pre>
  <p>When starting the thriftserver I use the provided <code>start-thriftserver.sh</code> command</p>