title: How to force Spark/Hive to create task_* directories with custom permissions
body: <p>I have the following problem. I will try to give as much details as possible, but if I miss anything that maybe useful to get this working, do not hesitate to ask.</p>
  
  <pre><code># spark-defaults:
  spark.sql.warehouse.dir = /mnt/data
  spark.hadoop.fs.permissions.umask-mode = 007
  
  # hive-site:
  &lt;property&gt;
    &lt;name&gt;hive.warehouse.subdir.inherit.perms&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;hive.metastore.execute.setugi&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
    &lt;description&gt;&lt;/description&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>I tried playing with the above hive-site settings with different combinations of those two, but the issue still persists.</p>
  
  <p>Spark cluster (standalone, master/workers/shuffle/thrift/history) is running as user <strong>spark</strong> (service account) and it is part of <strong>spark users</strong> group. No HDFS, but the filesystem is distributed and posix compliant (think of it as commercial HDFS), NFS v3 mounted. Hive metastore is in PostgreSQL 10.</p>
  
  <p>Spark Warehouse is here:</p>
  
  <pre><code># ls -l /mnt
  drwxrws--- 22 spark spark users 10240 Aug  9 09:31 data
  
  # umask
  0007
  </code></pre>
  
  <p>I run PySpark process as a <strong>user_1</strong> that is part of <strong>spark users</strong> group. The process creates database, creates table and writes data into the table.</p>
  
  <p>The process fails with the following exceptions:</p>
  
  <pre><code>18/08/09 09:31:42 ERROR FileFormatWriter: Aborting job null.
  java.io.IOException: Failed to rename 
  DeprecatedRawLocalFileStatus
  {path=file:/mnt/data/new.db/new_table/_temporary/0/ 
  task_20180809093142_0002_m_000000/
  part-00000-55f3fe5c-51c2-4a0f-9f0c-dc673f9967b3-c000.snappy.parquet;
  isDirectory=false; length=39330; replication=1; blocksize=33554432; 
  modification_time=1533821502000; access_time=0; owner=; group=; 
  permission=rw-rw-rw-; isSymlink=false} to 
  file:/mnt/data/new.db/new_table/
  part-00000-55f3fe5c-51c2-4a0f-9f0c-dc673f9967b3-c000.snappy.parquet
  at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths
  (FileOutputCommitter.java:415) at 
  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths
  (FileOutputCommitter.java:428) at 
  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal
  (FileOutputCommitter.java:362) at 
  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob
  (FileOutputCommitter.java:334) at 
  org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob
  (ParquetOutputCommitter.java:47) at 
  org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob
  (HadoopMapReduceCommitProtocol.scala:166) at 
  org.apache.spark.sql.execution.datasources.FileFormatWriter$.write
  (FileFormatWriter.scala:213) at 
  org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand
  .run (InsertIntoHadoopFsRelationCommand.scala:154) at 
  org.apache.spark.sql.execution.command.DataWritingCommandExec
  .sideEffectResult$lzycompute(commands.scala:104) at 
  org.apache.spark.sql.execution.command.DataWritingCommandExec
  .sideEffectResult(commands.scala:102) at 
  org.apache.spark.sql.execution.command.DataWritingCommandExec
  .executeCollect(commands.scala:115) at 
  org.apache.spark.sql.Dataset$$anonfun$6.apply
  (Dataset.scala:190) at 
  org.apache.spark.sql.Dataset$$anonfun$6.apply
  (Dataset.scala:190) at 
  org.apache.spark.sql.Dataset$$anonfun$52.apply
  (Dataset.scala:3254) at 
  org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId
  (SQLExecution.scala:77) at 
  org.apache.spark.sql.Dataset.withAction
  (Dataset.scala:3253) at 
  org.apache.spark.sql.Dataset.&lt;init&gt;
  (Dataset.scala:190) at 
  org.apache.spark.sql.Dataset$.ofRows
  (Dataset.scala:75) at 
  org.apache.spark.sql.SparkSession.sql
  (SparkSession.scala:641) at 
  sun.reflect.NativeMethodAccessorImpl.invoke0
  (Native Method) at 
  sun.reflect.NativeMethodAccessorImpl.invoke
  (NativeMethodAccessorImpl.java:62) at 
  sun.reflect.DelegatingMethodAccessorImpl.invoke
  (DelegatingMethodAccessorImpl.java:43) at 
  java.lang.reflect.Method.invoke
  (Method.java:498) at 
  py4j.reflection.MethodInvoker.invoke
  (MethodInvoker.java:244) at 
  py4j.reflection.ReflectionEngine.invoke
  (ReflectionEngine.java:357) at 
  py4j.Gateway.invoke(Gateway.java:282) at 
  py4j.commands.AbstractCommand.invokeMethod
  (AbstractCommand.java:132) at 
  py4j.commands.CallCommand.execute
  (CallCommand.java:79) at 
  py4j.GatewayConnection.run
  (GatewayConnection.java:238) at 
  java.lang.Thread.run
  (Thread.java:748) 
  18/08/09 09:31:42 WARN FileUtil: Failed to delete file or dir 
  [/mnt/data/new.db/new_table/
  _temporary/0/task_20180809093142_0002_m_000000/
  .part-00000-55f3fe5c-51c2-4a0f-9f0c-dc673f9967b3-c000.snappy.parquet.crc]: 
  it still exists.
  18/08/09 09:31:42 WARN FileUtil: Failed to delete file or dir 
  [/mnt/data/new.db/new_table/
  _temporary/0/task_20180809093142_0002_m_000000/
  part-00000-55f3fe5c-51c2-4a0f-9f0c-dc673f9967b3-c000.snappy.parquet]: 
  it still exists.
  </code></pre>
  
  <p>If fails to rename and delete files/directories. </p>
  
  <p>Directory structure:</p>
  
  <pre><code># ls -lR new.db/
  new.db/:
  total 4
  drwxrws--- 3 user_1 spark users 1024 Aug  9 09:31 new_table
  
  new.db/new_table:
  total 48
  -rw-rw---- 1 user_1 spark users 39330 Aug  9 09:31 part-00000-55f3fe5c-51c2-4a0f-9f0c-dc673f9967b3-c000.snappy.parquet
  drwxrws--- 3 user_1 spark users   512 Aug  9 09:31 _temporary
  
  new.db/new_table/_temporary:
  total 4
  drwxrws--- 3 user_1 spark users 512 Aug  9 09:31 0
  
  new.db/new_table/_temporary/0:
  total 4
  drwxr-sr-x 2 spark spark users 1024 Aug  9 09:31 task_20180809093142_0002_m_000000
  
  new.db/new_table/_temporary/0/task_20180809093142_0002_m_000000:
  total 44
  -rw-rw---- 1 spark spark users 39330 Aug  9 09:31 part-00000-55f3fe5c-51c2-4a0f-9f0c-dc673f9967b3-c000.snappy.parquet
  </code></pre>
  
  <p>As you can see the directories up to <strong>temporary/0</strong> (inclusive) are owned by <strong>user_1</strong>, but then <strong>task_</strong> directories inside <strong>temporary/0</strong> are owned by <strong>spark</strong> user. Moreover the umask that was used to create those <strong>task_</strong> directories was 022, and not the desired 007.</p>
  
  <p>If I can force <strong>spark</strong> user that creates those <strong>task_</strong> directories to actually use the correct umask, the issue would be resolved.</p>
  
  <p>I appreciate and pointers and advise.</p>