title: IPython- Py4JJavaError:
body: <p>I get a Py4JJavaError: when I try to create a data frame from rdd in pyspark. The problem is .createDataFrame() works in one ipython notebook and doesn't work in another. Strange. I have been trying to find out if there is synatx error I could nt fine one.This is my code:</p>
  
  <pre><code>grades.take(5)
  
  &gt;&gt;#output
  ['student_id,exam_1,exam_2,exam_3',
   '1,76,81,80',
   '2,90,92,96',
   '3,45,38,41',
   '4,87,88,82']
  
  header= grades.first()
  grades_=grades.filter(lambda x:x !=header)
  def function2(g):
      splt=g.split(",")
      result=[]
      for i in splt:
          result.append(int(i))
          return result
  new_grades=grades_.map(function2)
  new_grades.take(5)
  
  &gt;&gt;output
  [[1, 76, 81, 80],
   [2, 90, 92, 96],
   [3, 45, 38, 41],
   [4, 87, 88, 82],
   [5, 69, 65, 66]]
  
  sqlContext = SQLContext(sc)
  grades_df =sqlContext.createDataFrame(new_grades, ['student_id','exam_1','exam_2','exam_3'])
  
  ###the above line gives the following error. It worked fine for the previous code i
  wrote in another ipython notebook. Both the notebooks are in the same location
  </code></pre>
  
  <hr>
  
  <pre><code>Py4JJavaError                             Traceback (most recent call last)
  &lt;ipython-input-266-919ed44c544a&gt; in &lt;module&gt;()
        1 #rs_df =sqlContext.createDataFrame(rs, ['Date','Time','Store_Name','Item', 'Cost', 'Payment_Type'])
  ----&gt; 2 grades_df =sqlContext.createDataFrame(new_grades, ['student_id','exam_1','exam_2','exam_3'])
  
  c:\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\context.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
      307         Py4JJavaError: ...
      308         """
  --&gt; 309         return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
      310 
      311     @since(1.3)
  c:\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
      524             rdd, schema = self._createFromLocal(map(prepare, data), schema)
      525         jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
  --&gt; 526         jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
      527         df = DataFrame(jdf, self._wrapped)
      528         df._schema = schema
  
  c:\spark-2.0.1-bin-hadoop2.7\python\lib\py4j-0.10.3-src.zip\py4j\java_gateway.py in __call__(self, *args)
     1131         answer = self.gateway_client.send_command(command)
     1132         return_value = get_return_value(
  -&gt; 1133             answer, self.gateway_client, self.target_id, self.name)
     1134 
     1135         for temp_arg in temp_args:
  
  c:\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\utils.py in deco(*a, **kw)
       61     def deco(*a, **kw):
       62         try:
  ---&gt; 63             return f(*a, **kw)
       64         except py4j.protocol.Py4JJavaError as e:
       65             s = e.java_exception.toString()
  
  c:\spark-2.0.1-bin-hadoop2.7\python\lib\py4j-0.10.3-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
      317                 raise Py4JJavaError(
      318                     "An error occurred while calling {0}{1}{2}.\n".
  --&gt; 319                     format(target_id, ".", name), value)
      320             else:
      321                 raise Py4JError(
  
  Py4JJavaError: An error occurred while calling o2069.applySchemaToPythonRDD.
  : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
      at org.apache.spark.sql.hive.client.HiveClientImpl.&lt;init&gt;(HiveClientImpl.scala:189)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)
      at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50)
      at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48)
      at org.apache.spark.sql.hive.HiveSessionState$$anon$1.&lt;init&gt;(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62)
      at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
      at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:666)
      at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:656)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:86)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
      ... 32 more
  Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
      ... 38 more
  Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
  java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@324f83d6, see the next exception for details.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.&lt;init&gt;(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
      at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
      at java.sql.DriverManager.getConnection(DriverManager.java:664)
      at java.sql.DriverManager.getConnection(DriverManager.java:208)
      at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
      at com.jolbox.bonecp.BoneCP.&lt;init&gt;(BoneCP.java:416)
      at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
      at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:501)
      at org.datanucleus.store.rdbms.RDBMSStoreManager.&lt;init&gt;(RDBMSStoreManager.java:298)
  
  c:\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
      524             rdd, schema = self._createFromLocal(map(prepare, data), schema)
      525         jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
  --&gt; 526         jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
      527         df = DataFrame(jdf, self._wrapped)
      528         df._schema = schema
  
  c:\spark-2.0.1-bin-hadoop2.7\python\lib\py4j-0.10.3-src.zip\py4j\java_gateway.py in __call__(self, *args)
     1131         answer = self.gateway_client.send_command(command)
     1132         return_value = get_return_value(
  -&gt; 1133             answer, self.gateway_client, self.target_id, self.name)
     1134 
     1135         for temp_arg in temp_args:
  
  c:\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\utils.py in deco(*a, **kw)
       61     def deco(*a, **kw):
       62         try:
  ---&gt; 63             return f(*a, **kw)
       64         except py4j.protocol.Py4JJavaError as e:
       65             s = e.java_exception.toString()
  
  c:\spark-2.0.1-bin-hadoop2.7\python\lib\py4j-0.10.3-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
      317                 raise Py4JJavaError(
      318                     "An error occurred while calling {0}{1}{2}.\n".
  --&gt; 319                     format(target_id, ".", name), value)
      320             else:
      321                 raise Py4JError(
  
  Py4JJavaError: An error occurred while calling o2069.applySchemaToPythonRDD.
  : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
      at org.apache.spark.sql.hive.client.HiveClientImpl.&lt;init&gt;(HiveClientImpl.scala:189)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)
      at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50)
      at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48)
      at org.apache.spark.sql.hive.HiveSessionState$$anon$1.&lt;init&gt;(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62)
      at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
      at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:666)
      at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:656)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:86)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
      ... 32 more
  Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
      ... 38 more
  Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
  java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@324f83d6, see the next exception for details.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.&lt;init&gt;(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
      at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
      at java.sql.DriverManager.getConnection(DriverManager.java:664)
      at java.sql.DriverManager.getConnection(DriverManager.java:208)
      at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
      at com.jolbox.bonecp.BoneCP.&lt;init&gt;(BoneCP.java:416)
      at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
      at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:501)
      at org.datanucleus.store.rdbms.RDBMSStoreManager.&lt;init&gt;(RDBMSStoreManager.java:298)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
      at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
      at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
      at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
      at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
      at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
      at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.&lt;init&gt;(RawStoreProxy.java:57)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:624)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:66)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
      at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:199)
      at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.&lt;init&gt;(SessionHiveMetaStoreClient.java:74)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:86)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
      at org.apache.spark.sql.hive.client.HiveClientImpl.&lt;init&gt;(HiveClientImpl.scala:189)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)
      at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50)
      at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48)
      at org.apache.spark.sql.hive.HiveSessionState$$anon$1.&lt;init&gt;(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62)
      at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
      at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:666)
      at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:656)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@324f83d6, see the next exception for details.
      at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
  </code></pre>