title: spark databricks cluster noclassdeffounderror during insert
body: <p>Experimenting with Databricks Spark cluster. When creating a table in a Hive database, I get the following error the first time.</p>
  
  <pre><code>19/06/18 21:34:17 ERROR SparkExecuteStatementOperation: Error running hive query: 
  org.apache.hive.service.cli.HiveSQLException: java.lang.NoClassDefFoundError: org/joda/time/ReadWritableInstant
      at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:296)
      at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2$$anonfun$run$2.apply$mcV$sp(SparkExecuteStatementOperation.scala:182)
      at org.apache.spark.sql.hive.thriftserver.server.SparkSQLUtils$class.withLocalProperties(SparkSQLOperationManager.scala:190)
  </code></pre>
  
  <p>On subsequent attempts to create the same table (without restarting the cluster), I get this...</p>
  
  <pre><code>org.apache.hive.service.cli.HiveSQLException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyPrimitiveObjectInspectorFactory
      at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:296)
      at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2$$anonfun$run$2.apply$mcV$sp(SparkExecuteStatementOperation.scala:182)
      at org.apache.spark.sql.hive.thriftserver.server.SparkSQLUtils$class.withLocalProperties(SparkSQLOperationManager.scala:190)
      at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:44)
  </code></pre>
  
  <p>From beeline (client), I get the following errors.... essentially the same thing.</p>
  
  <pre><code>13: jdbc:spark://dbc-e1ececb9-10d2.cloud.data&gt; create table test_dnax_db.sample2 (name2 string);
  Error: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: java.lang.NoClassDefFoundError: org/joda/time/ReadWritableInstant, Query: create table test_dnax_db.sample2 (name2 string). (state=HY000,code=500051)
  13: jdbc:spark://dbc-e1ececb9-10d2.cloud.data&gt; create table test_dnax_db.sample2 (name2 string);
  Error: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyPrimitiveObjectInspectorFactory, Query: create table test_dnax_db.sample2 (name2 string). (state=HY000,code=500051)
  </code></pre>
  
  <p>I've tried uploading dependent joda-time jars and serde jars using databricks's <code>libraries</code> feature. Also, I've set the spark property <code>spark.driver.extraClassPath</code> (given error comes from Spark driver and not the workers). Neither helps. I do see the dependent jars available on the hosts /databricks/hive and /databricks/jars folder. </p>
  
  <p>I've also tried to set environment variables like <code>HADOOP_CLASSPATH</code>, without much luck.</p>
  
  <p>Databricks forums are notoriously useless as they are not curated at all (compared to splunk or similar commercial products).</p>
  
  <p>Any suggestions welcome.</p>
  
  <p>I can successfully create database using <code>location</code> keyword as well query from an existing table in the metastore.</p>
  
  <p>EDIT:</p>
  
  <p>I suspect the SparkExecuteStatementOperation (thrift entry class to sql execution in spark cluster, running on the driver) might be using a different classloader different from the application. I added this in my application class static block, which I know gets initialized and I see no ClassNotFoundException i.e. jar is available to application. But underlying driver does not see the relevant jar.</p>
  
  <pre><code>static {
          try {
              Class&lt;?&gt; aClass = Class.forName("org.joda.time.ReadWritableInstant");
              }
          } catch (ClassNotFoundException e) {
              LOG.warn("Unable to find ReadWritableInstant class", e);
          }
  }
  </code></pre>