title: Spark Can't Read Local File
body: <p>I have a local file available on all Spark nodes in an EMR cluster with the following permissions:</p>
  
  <pre><code>-rw-rw---- 1 test_user test_group 30 Jun 21 14:20 /tmp/foo_test
  </code></pre>
  
  <p>I'm running the cluster as <code>ec2-user</code>, using the yarn scheduler.  In order for Spark/Yarn to have access to the file, I added <code>test_group</code> as a secondary group of the <code>yarn</code> user on all the nodes.</p>
  
  <pre><code>$ sudo -u yarn groups
    yarn hadoop test_group
  </code></pre>
  
  <p>In spark-shell, I get the following error reading the file:</p>
  
  <pre><code>scala&gt; val rdd = sc.textFile("file:///tmp/foo_test")
      org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0:
  java.io.FileNotFoundException: /tmp/foo_test (Permission denied)
      at java.io.FileInputStream.open(Native Method)
      at java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:146)
      at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.&lt;init&gt;(RawLocalFileSystem.java:111)
      at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:207)
      at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.&lt;init&gt;(ChecksumFileSystem.java:141)
      at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
      at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:771)
      at org.apache.hadoop.mapred.LineRecordReader.&lt;init&gt;(LineRecordReader.java:109)
      at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
      at org.apache.spark.rdd.HadoopRDD$$anon$1.&lt;init&gt;(HadoopRDD.scala:237)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>How do I read files with group-level permissions on EMR Spark?</p>