title: FileNotFoundException on _temporary/0 directory when saving Parquet files
body: <p>Using Python on an Azure HDInsight cluster, we are saving Spark dataframes as Parquet files to an Azure Data Lake Storage Gen2, using the following code:</p>
  <pre class="lang-py prettyprint-override"><code>df.write.parquet('abfs://my_dwh_container@my_storage_account.dfs.core.windows.net/mypath, 'overwrite', compression='snappy')
  </code></pre>
  <p>Often this works, but when we recently upgraded our cluster to run more scripts at the same time (around ten to fifteen) we consistently get the following exception for a varying small fraction of the scripts:</p>
  <blockquote>
  <p>Py4JJavaError: An error occurred while calling o2232.parquet. :
  java.io.FileNotFoundException: Operation failed: &quot;The specified path
  does not exist.&quot;, 404, PUT,
  <a href="https://my_storage_account.dfs.core.windows.net/mypath/_temporary/0?resource=directory&amp;timeout=90" rel="nofollow noreferrer">https://my_storage_account.dfs.core.windows.net/mypath/_temporary/0?resource=directory&amp;timeout=90</a>,
  PathNotFound, &quot;The specified path does not exist.&quot;</p>
  </blockquote>
  <p>I think all the Spark jobs and tasks actually succeed, also the one that saves the table, but then the Python script exits with the exception.</p>
  <hr />
  <h2>Background information</h2>
  <p>We are using Spark 2.4.5.4.1.1.2. Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_265, Hadoop 3.1.2.4.1.1.2</p>
  <p>Stacktrace:</p>
  <pre><code>  File &quot;/usr/hdp/current/spark2-client/python/pyspark/sql/readwriter.py&quot;, line 843, in parquet
      df_to_save.write.parquet(blob_path, mode, compression='snappy')
      self._jwrite.parquet(path)
    File &quot;/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py&quot;, line 1257, in __call__
      answer, self.gateway_client, self.target_id, self.name)
    File &quot;/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py&quot;, line 63, in deco
      return f(*a, **kw)
    File &quot;/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py&quot;, line 328, in get_return_value
      format(target_id, &quot;.&quot;, name), value)
  py4j.protocol.Py4JJavaError: An error occurred while calling o2232.parquet.
  : java.io.FileNotFoundException: Operation failed: &quot;The specified path does not exist.&quot;, 404, PUT, https://my_dwh_container@my_storage_account.dfs.core.windows.net/mypath/_temporary/0?resource=directory&amp;timeout=90, PathNotFound, &quot;The specified path does not exist. RequestId:1870ec49-e01f-0101-72f8-f260fe000000 Time:2021-12-17T03:42:35.8434071Z&quot;
      at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1178)
      at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.mkdirs(AzureBlobFileSystem.java:477)
      at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2288)
      at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:382)
      at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:162)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:139)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
      at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
      at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
      at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
      at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
      at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
      at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
      at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
      at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
      at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
      at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
      at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
      at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
      at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
      at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
      at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:566)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  <p>Log:</p>
  <pre><code>21/12/17 03:42:02 INFO DAGScheduler [Thread-11]: Job 2 finished: saveAsTable at NativeMethodAccessorImpl.java:0, took 1.120535 s
  21/12/17 03:42:02 INFO FileFormatWriter [Thread-11]: Write Job 11fc45a5-d398-4f9a-8350-f928c3722886 committed.
  21/12/17 03:42:02 INFO FileFormatWriter [Thread-11]: Finished processing stats for write job 11fc45a5-d398-4f9a-8350-f928c3722886.
  (...)
  21/12/17 03:42:05 INFO ParquetFileFormat [Thread-11]: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter
  21/12/17 03:42:05 INFO FileOutputCommitter [Thread-11]: File Output Committer Algorithm version is 2
  21/12/17 03:42:05 INFO FileOutputCommitter [Thread-11]: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false, move _temporary folders into Trash: false
  21/12/17 03:42:05 INFO SQLHadoopMapReduceCommitProtocol [Thread-11]: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
  21/12/17 03:42:05 INFO FileOutputCommitter [Thread-11]: File Output Committer Algorithm version is 2
  21/12/17 03:42:05 INFO FileOutputCommitter [Thread-11]: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false, move _temporary folders into Trash: false
  21/12/17 03:42:05 INFO SQLHadoopMapReduceCommitProtocol [Thread-11]: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
  21/12/17 03:42:28 ERROR ApplicationMaster [Driver]: User application exited with status 1
  21/12/17 03:42:28 INFO ApplicationMaster [Driver]: Final app status: FAILED, exitCode: 1, (reason: User application exited with status 1)
  </code></pre>
  <p>There is also another version of this exception which does occur in a Spark task which then fails, but Spark automatically restarts the failed task and usually it succeeds then. In some cases, the AM will report the app as failed, but I don't understand why because all jobs succeeded.</p>
  <p><strong>Possible causes</strong></p>
  <p>As seen in <a href="https://stackoverflow.com/questions/46882683/spark-temporary-creation-reason">Spark _temporary creation reason</a> I would expect that the _temporary directory will not be moved until all tasks are done.
  Looking at the stacktrace, it happens in AzureBlobFileSystem.mkdirs, which suggests to me that it's trying to create subdirectories somewhere under <code>_temporary/0</code>, but it cannot find the <code>0</code> directory. I'm not sure if the <code>_temporary</code> directory exists at that point.</p>
  <p><strong>Related questions</strong></p>
  <ul>
  <li><a href="https://issues.apache.org/jira/browse/SPARK-2984" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-2984</a> It does sound similar, but I don't see tasks being restarted because they take long, and this should have been fixed a long time ago anyway. I'm not completely sure if speculative execution is visible in the Spark UI though.</li>
  <li><a href="https://stackoverflow.com/questions/51603404/saving-dataframe-to-local-file-system-results-in-empty-results">Saving dataframe to local file system results in empty results</a> We are not saving to any local file system (even though the error message does say https, the stacktrace shows AzureBlobFileSystem.</li>
  <li><a href="https://stackoverflow.com/questions/52089208/spark-dataframe-write-to-csv-creates-temporary-directory-file-in-standalone-clu">Spark Dataframe Write to CSV creates _temporary directory file in Standalone Cluster Mode</a> We <em>are</em> using HDFS and also file output committer 2</li>
  <li><a href="https://stackoverflow.com/questions/38964736/multiple-spark-jobs-appending-parquet-data-to-same-base-path-with-partitioning">Multiple spark jobs appending parquet data to same base path with partitioning</a> I don't think two jobs make use of the same directory here</li>
  <li><a href="https://community.datastax.com/questions/3082/while-writing-to-parquet-file-on-hdfs-throwing-fil.html" rel="nofollow noreferrer">https://community.datastax.com/questions/3082/while-writing-to-parquet-file-on-hdfs-throwing-fil.html</a> I don't think this is a permissions issue, as most of the time it does work.</li>
  <li><a href="https://stackoverflow.com/questions/42822483/extremely-slow-s3-write-times-from-emr-spark">Extremely slow S3 write times from EMR/ Spark</a> We don't have any problems regarding slow renaming, as far as I know (the files aren't very large anyway). I think it fails before renaming, so a zero-rename committer wouldn't help here?</li>
  <li><a href="https://support.huaweicloud.com/intl/en-us/trouble-mrs/mrs_03_0084.html" rel="nofollow noreferrer">https://support.huaweicloud.com/intl/en-us/trouble-mrs/mrs_03_0084.html</a> Suggests to look in the namenode audit log of hdfs, but haven't yet found it.</li>
  <li><a href="https://github.com/apache/hadoop/blob/b7d2135f6f5cea7cf5d5fc5a2090fc5d8596969e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java#L355" rel="nofollow noreferrer">https://github.com/apache/hadoop/blob/b7d2135f6f5cea7cf5d5fc5a2090fc5d8596969e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java#L355</a> Since the stacktrace shows it fails at mkdirs, I'm guessing the <code>_temporary</code> itself doesn't exist, but I don't understand why mkdirs doesn't create it. But I don't think AzureBlobFileSystem is open source?</li>
  <li>I did find <a href="https://jar-download.com/artifacts/org.apache.hadoop/hadoop-azure/3.2.0/source-code/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java" rel="nofollow noreferrer">some version of AzureBlobFileSystem.java</a> but based on the stacktrace it would go to checkException with a <code>PATH_ALREADY_EXISTS</code> flag which doesn't make sense to me.</li>
  </ul>
  <p>Possible options to try:</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/59261265/pyspark-dataframe-write-parquet-without-deleting-temporary-folder">Pyspark dataframe write parquet without deleting /_temporary folder</a> What we could try, is first saving to a different HDFS and then copy the final files. I'm not sure why it would help, because we're already saving to HDFS (well, an extension of it, ADFS).</li>
  <li><a href="https://community.cloudera.com/t5/Support-Questions/How-to-change-Spark-temporary-directory-when-writing-data/td-p/237389" rel="nofollow noreferrer">https://community.cloudera.com/t5/Support-Questions/How-to-change-Spark-temporary-directory-when-writing-data/td-p/237389</a> We could try using append and delete the files ourselves.</li>
  <li><a href="https://stackoverflow.com/questions/55595930/change-spark-temporary-directory-path">Change spark _temporary directory path</a> Using our own FileOutputCommitter sounds overkill for this problem</li>
  </ul>