title: Unable to run pyspark on local windows environment: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativei
body: <p>I'm trying to create local spark environment in Windows 11 with python. <br />
  I am using python 3.9 and spark version 3.2.1.
  I have set my environmental variables to:</p>
  <pre><code>PYTHONPATH = C:\Users\nina\AppData\Local\Programs\Python\Python39
  SPARK_HOME = C:\spark\spark-3.2.1-bin-hadoop3.2
  HADOOP_HOME = %SPARK_HOME%\hadoop
  </code></pre>
  <p>And I've added <code>%SPARK_HOME%\bin</code>, <code>%PYTHONPATH%</code> and <code>%HADOOP_HOME%\bin</code> to my path.<br />
  I've downloaded <code>hadoop.dll</code> and <code>winutils.exe</code> from <a href="https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin" rel="nofollow noreferrer">https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin</a> <br />
  I've also tried to use these files from here <a href="https://github.com/cdarlint/winutils/blob/master/hadoop-3.2.2/bin/hadoop.dll" rel="nofollow noreferrer">https://github.com/cdarlint/winutils/blob/master/hadoop-3.2.2/bin/hadoop.dll</a>
  But I still get the same error:</p>
  <pre><code>Warning: Ignoring non-Spark config property: derby.system.home
  Warning: Ignoring non-Spark config property: derby.stream.error.file
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  Setting default log level to &quot;WARN&quot;.
  To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
  22/03/16 10:41:20 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
  22/03/16 10:41:20 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
  22/03/16 10:41:25 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
  22/03/16 10:41:25 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore UNKNOWN@192.168.33.80
  22/03/16 10:41:25 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
  22/03/16 10:41:26 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
  22/03/16 10:41:26 WARN ObjectStore: Failed to get database test_db, returning NoSuchObjectException
  
  Failure
  Traceback (most recent call last):
    File &quot;C:\Users\nina\AppData\Local\Programs\Python\Python39\lib\unittest\suite.py&quot;, line 166, in _handleClassSetUp
      setUpClass()
       cls.spark.sql(f&quot;CREATE DATABASE IF NOT EXISTS {cls.db_name}&quot;)
    File &quot;C:\Users\nina\PycharmProjects\raw_logs\venv\lib\site-packages\pyspark\sql\session.py&quot;, line 723, in sql
      return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
    File &quot;C:\Users\nina\PycharmProjects\raw_logs\venv\lib\site-packages\py4j\java_gateway.py&quot;, line 1321, in __call__
      return_value = get_return_value(
    File &quot;C:\Users\nina\PycharmProjects\raw_logs\venv\lib\site-packages\pyspark\sql\utils.py&quot;, line 111, in deco
      return f(*a, **kw)
    File &quot;C:\Users\nina\PycharmProjects\raw_logs\venv\lib\site-packages\py4j\protocol.py&quot;, line 326, in get_return_value
      raise Py4JJavaError(
  py4j.protocol.Py4JJavaError: An error occurred while calling o35.sql.
  : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
      at org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Native Method)
      at org.apache.hadoop.io.nativeio.NativeIO$POSIX.getStat(NativeIO.java:608)
      at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNativeIO(RawLocalFileSystem.java:934)
      at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:848)
      at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getGroup(RawLocalFileSystem.java:832)
      at org.apache.hadoop.hive.io.HdfsUtils.setFullFileStatus(HdfsUtils.java:102)
      at org.apache.hadoop.hive.io.HdfsUtils.setFullFileStatus(HdfsUtils.java:94)
      at org.apache.hadoop.hive.io.HdfsUtils.setFullFileStatus(HdfsUtils.java:77)
      at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:544)
      at org.apache.hadoop.hive.metastore.Warehouse.mkdirs(Warehouse.java:194)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:880)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:939)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
      at com.sun.proxy.$Proxy17.create_database(Unknown Source)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:725)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
      at com.sun.proxy.$Proxy18.createDatabase(Unknown Source)
      at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:434)
      at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createDatabase$1(HiveClientImpl.scala:347)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
      at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:305)
      at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:236)
      at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:235)
      at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:285)
      at org.apache.spark.sql.hive.client.HiveClientImpl.createDatabase(HiveClientImpl.scala:345)
      at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createDatabase$1(HiveExternalCatalog.scala:193)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
      at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:102)
      at org.apache.spark.sql.hive.HiveExternalCatalog.createDatabase(HiveExternalCatalog.scala:193)
      at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createDatabase(ExternalCatalogWithListener.scala:47)
      at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:251)
      at org.apache.spark.sql.execution.command.CreateDatabaseCommand.run(ddl.scala:83)
      at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
      at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
      at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
      at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
      at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
      at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
      at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
      at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
      at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
      at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
      at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
      at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
      at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
      at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
      at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
      at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
      at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
      at org.apache.spark.sql.Dataset.&lt;init&gt;(Dataset.scala:219)
      at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
      at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
      at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
      at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
      at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
      at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
      at java.lang.Thread.run(Thread.java:750)
  </code></pre>
  <p>I setup my spark as following:</p>
  <pre><code>from pyspark.sql import SparkSession
  import os
  
  os.environ['PYSPARK_PYTHON'] = sys.executable
  os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
  
  derby_sys_dir = os.path.join(os.path.expanduser('~'), 'derby')
  
  spark = SparkSession.builder.master(&quot;local[*]&quot;) \
             .config(&quot;spark.hadoop.hive.metastore.warehouse.dir&quot;, derby_sys_dir) \
             .config(&quot;derby.system.home&quot;, derby_sys_dir) \
             .config(&quot;derby.stream.error.file&quot;, derby_sys_dir) \
             .config(&quot;spark.driver.host&quot;, &quot;localhost&quot;) \
             .config(&quot;spark.sql.warehouse.dir&quot;, derby_sys_dir) \
             .enableHiveSupport().appName(&quot;TestXMLLogsProcessor&quot;).getOrCreate()
  </code></pre>
  <p>What am I missing?</p>