title: Spark no such field METASTORE_CLIENT_FACTORY_CLASS
body: <p>I am trying to query a hive table using spark in Java. My hive tables are in an EMR cluster 5.12. Spark version is 2.2.1 and Hive 2.3.2.</p>
  
  <p>When I ssh into the machine and I connect to the spark-shell I am able to query the hive tables with no issues. </p>
  
  <p>But when I try to query using a custom jar then I get the following exception:</p>
  
  <pre><code>java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
      at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1067)
      at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:142)
      at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:141)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:141)
      at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:138)
      at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:637)
      at package.Session.executeQuery(Session.java:48)
      at com.etl.cli.ETL.main(ETL.java:16)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
  Caused by: org.apache.spark.sql.AnalysisException: java.lang.NoSuchFieldError: METASTORE_CLIENT_FACTORY_CLASS;
          at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
          at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
          at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:105)
          at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:35)
          at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289)
          at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1064)
          ... 16 more
  Caused by: java.lang.NoSuchFieldError: METASTORE_CLIENT_FACTORY_CLASS
          at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClientFactory(Hive.java:3011)
          at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3006)
          at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3042)
          at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1235)
          at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:175)
          at org.apache.hadoop.hive.ql.metadata.Hive.&lt;clinit&gt;(Hive.java:167)
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
          at org.apache.spark.sql.hive.client.HiveClientImpl.&lt;init&gt;(HiveClientImpl.scala:191)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
          at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:362)
          at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:266)
          at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
          at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
          at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:195)
          at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195)
          at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195)
          at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
          ... 25 more
  </code></pre>
  
  <p>My pom looks like this :</p>
  
  <pre><code>  &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-core_2.11&lt;/artifactId&gt;
          &lt;version&gt;2.2.1&lt;/version&gt;
      &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
          &lt;version&gt;2.2.1&lt;/version&gt;
      &lt;/dependency&gt;
  </code></pre>
  
  <p>Instantiating the session:</p>
  
  <pre><code> SparkConf conf = new SparkConf();
          conf.set("spark.sql.warehouse.dir", "/user/hive/warehouse");
          conf.set("hive.metastore.client.factory.class",
                  "org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.Client.Factory");
  
          SparkSession sparkSession = SparkSession.builder()
                  .appName("Spark Application")
                  .master("yarn-cluster")
                  .config(conf)
                  .enableHiveSupport()
                  .getOrCreate();
  </code></pre>
  
  <p>Any ideas what might be wrong?</p>
  
  <p>Thanks</p>