title: Apache Spark 2.3.1 with Hive metastore 3.1.0
body: <p>We have upgraded HDP cluster to 3.1.1.3.0.1.0-187 and have discovered:</p>
  
  <ol>
  <li>Hive has a new metastore location </li>
  <li>Spark can't see Hive databases</li>
  </ol>
  
  <p>In fact we see:</p>
  
  <pre><code>org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database ... not found
  </code></pre>
  
  <p>Could you help me understanding what has happened and how to solve this?</p>
  
  <p><strong>Update:</strong></p>
  
  <p>Configuration:</p>
  
  <blockquote>
    <p>(spark.sql.warehouse.dir,/warehouse/tablespace/external/hive/)
    (spark.admin.acls,)
    (spark.yarn.dist.files,file:///opt/folder/config.yml,file:///opt/jdk1.8.0_172/jre/lib/security/cacerts)
    (spark.history.kerberos.keytab,/etc/security/keytabs/spark.service.keytab)
    (spark.io.compression.lz4.blockSize,128kb)
    (spark.executor.extraJavaOptions,-Djavax.net.ssl.trustStore=cacerts)
    (spark.history.fs.logDirectory,hdfs:///spark2-history/)
    (spark.io.encryption.keygen.algorithm,HmacSHA1)
    (spark.sql.autoBroadcastJoinThreshold,26214400)
    (spark.eventLog.enabled,true) (spark.shuffle.service.enabled,true)
    (spark.driver.extraLibraryPath,/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64)
    (spark.ssl.keyStore,/etc/security/serverKeys/server-keystore.jks)
    (spark.yarn.queue,default)
    (spark.jars,file:/opt/folder/component-assembly-0.1.0-SNAPSHOT.jar)
    (spark.ssl.enabled,true) (spark.sql.orc.filterPushdown,true)
    (spark.shuffle.unsafe.file.output.buffer,5m)
    (spark.yarn.historyServer.address,master2.env.project:18481)
    (spark.ssl.trustStore,/etc/security/clientKeys/all.jks)
    (spark.app.name,com.company.env.component.MyClass)
    (spark.sql.hive.metastore.jars,/usr/hdp/current/spark2-client/standalone-metastore/*)
    (spark.io.encryption.keySizeBits,128) (spark.driver.memory,2g)
    (spark.executor.instances,10)
    (spark.history.kerberos.principal,spark/edge.env.project@ENV.PROJECT)
    (spark.unsafe.sorter.spill.reader.buffer.size,1m)
    (spark.ssl.keyPassword,*********(redacted))
    (spark.ssl.keyStorePassword,*********(redacted))
    (spark.history.fs.cleaner.enabled,true)
    (spark.shuffle.io.serverThreads,128)
    (spark.sql.hive.convertMetastoreOrc,true)
    (spark.submit.deployMode,client) (spark.sql.orc.char.enabled,true)
    (spark.master,yarn) (spark.authenticate.enableSaslEncryption,true)
    (spark.history.fs.cleaner.interval,7d) (spark.authenticate,true)
    (spark.history.fs.cleaner.maxAge,90d)
    (spark.history.ui.acls.enable,true) (spark.acls.enable,true)
    (spark.history.provider,org.apache.spark.deploy.history.FsHistoryProvider)
    (spark.executor.extraLibraryPath,/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64)
    (spark.executor.memory,2g) (spark.io.encryption.enabled,true)
    (spark.shuffle.file.buffer,1m)
    (spark.eventLog.dir,hdfs:///spark2-history/) (spark.ssl.protocol,TLS)
    (spark.dynamicAllocation.enabled,true) (spark.executor.cores,3)
    (spark.history.ui.port,18081)
    (spark.sql.statistics.fallBackToHdfs,true)
    (spark.repl.local.jars,file:///opt/folder/postgresql-42.2.2.jar,file:///opt/folder/ojdbc6.jar)
    (spark.ssl.trustStorePassword,*********(redacted))
    (spark.history.ui.admin.acls,) (spark.history.kerberos.enabled,true)
    (spark.shuffle.io.backLog,8192) (spark.sql.orc.impl,native)
    (spark.ssl.enabledAlgorithms,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA)
    (spark.sql.orc.enabled,true)
    (spark.yarn.dist.jars,file:///opt/folder/postgresql-42.2.2.jar,file:///opt/folder/ojdbc6.jar)
    (spark.sql.hive.metastore.version,3.0)</p>
  </blockquote>
  
  <p>And from hive-site.xml:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;hive.metastore.warehouse.dir&lt;/name&gt;
    &lt;value&gt;/warehouse/tablespace/managed/hive&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>Code looks like:</p>
  
  <pre><code>val spark = SparkSession
    .builder()
    .appName(getClass.getSimpleName)
    .enableHiveSupport()
    .getOrCreate()
  ...
  dataFrame.write
    .format("orc")
    .options(Map("spark.sql.hive.convertMetastoreOrc" -&gt; true.toString))
    .mode(SaveMode.Append)
    .saveAsTable("name")
  </code></pre>
  
  <p>Spark-submit:</p>
  
  <pre><code>    --master yarn \
      --deploy-mode client \
      --driver-memory 2g \
      --driver-cores 4 \
      --executor-memory 2g \
      --num-executors 10 \
      --executor-cores 3 \
      --conf "spark.dynamicAllocation.enabled=true" \
      --conf "spark.shuffle.service.enabled=true" \
      --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=cacerts" \
      --conf "spark.sql.warehouse.dir=/warehouse/tablespace/external/hive/" \
      --jars postgresql-42.2.2.jar,ojdbc6.jar \
      --files config.yml,/opt/jdk1.8.0_172/jre/lib/security/cacerts \
      --verbose \
      component-assembly-0.1.0-SNAPSHOT.jar \
  </code></pre>