title: Spark Hive reporting ClassNotFoundException: com.ibm.biginsights.bigsql.sync.BIEventListener
body: <p>I'm attempting to run a pyspark script on BigInsights on Cloud 4.2 Enterprise that accesses a Hive table.</p>
  
  <p>First I create the hive table:</p>
  
  <pre><code>[biadmin@bi4c-xxxxx-mastermanager ~]$ hive
  hive&gt; CREATE TABLE pokes (foo INT, bar STRING);
  OK
  Time taken: 2.147 seconds
  hive&gt; LOAD DATA LOCAL INPATH '/usr/iop/4.2.0.0/hive/doc/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
  Loading data to table default.pokes
  Table default.pokes stats: [numFiles=1, numRows=0, totalSize=5812, rawDataSize=0]
  OK
  Time taken: 0.49 seconds
  hive&gt; 
  </code></pre>
  
  <p>Then I create a simple pyspark script:</p>
  
  <pre><code>[biadmin@bi4c-xxxxxx-mastermanager ~]$ cat test_pokes.py
  from pyspark import SparkContext
  
  sc = SparkContext()
  
  from pyspark.sql import HiveContext
  hc = HiveContext(sc)
  
  pokesRdd = hc.sql('select * from pokes')
  print( pokesRdd.collect() )
  </code></pre>
  
  <p>I attempt to execute with:</p>
  
  <pre><code>[biadmin@bi4c-xxxxxx-mastermanager ~]$ spark-submit \
      --master yarn-cluster \
      --deploy-mode cluster \
      --jars /usr/iop/4.2.0.0/hive/lib/datanucleus-api-jdo-3.2.6.jar, \
             /usr/iop/4.2.0.0/hive/lib/datanucleus-core-3.2.10.jar, \
             /usr/iop/4.2.0.0/hive/lib/datanucleus-rdbms-3.2.9.jar \
      --files /usr/iop/4.2.0.0/hive/conf/hive-site.xml \
      test_pokes.py
  </code></pre>
  
  <p>However, I encounter the error:</p>
  
  <pre><code>You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly
  Traceback (most recent call last):
    File "test_pokes.py", line 8, in &lt;module&gt;
      pokesRdd = hc.sql('select * from pokes')
    File "/disk2/local/usercache/biadmin/appcache/application_1477084339086_0485/container_e09_1477084339086_0485_02_000001/pyspark.zip/pyspark/sql/context.py", line 580, in sql
    ...
    File /container_e09_1477084339086_0485_02_000001/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
  py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.
  : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
      at 
      ...
      ...
  Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
      at 
      ...
      ... 27 more
  Caused by: MetaException(message:Failed to instantiate listener named: com.ibm.biginsights.bigsql.sync.BIEventListener, reason: java.lang.ClassNotFoundException: com.ibm.biginsights.bigsql.sync.BIEventListener)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreListeners(MetaStoreUtils.java:1478)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:481)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:66)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
      at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:199)
      at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.&lt;init&gt;(SessionHiveMetaStoreClient.java:74)
      ... 32 more
  </code></pre>
  
  <p>See also previous errors related to this issue:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/41262756/hive-spark-yarn-cluster-job-fails-with-classnotfoundexception-org-datanucleus/41263218#41263218">hive spark yarn-cluster job fails with: &quot;ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory&quot;</a></li>
  <li><a href="https://stackoverflow.com/questions/41263670/spark-hive-reporting-pyspark-sql-utils-analysisexception-utable-not-found-xxx">Spark Hive reporting pyspark.sql.utils.AnalysisException: u&#39;Table not found: XXX&#39; when run on yarn cluster</a></li>
  </ul>