title: How to avoid "Not a file" exceptions when reading from HDFS with spark
body: <p>I copy a tree of files from S3 to HDFS with <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html" rel="noreferrer" title="S3DistCp">S3DistCP</a> in an initial EMR step. <code>hdfs dfs -ls -R hdfs:///data_dir</code> shows the expected files, which look something like:</p>
  
  <pre><code>/data_dir/year=2015/
  /data_dir/year=2015/month=01/
  /data_dir/year=2015/month=01/day=01/
  /data_dir/year=2015/month=01/day=01/data01.12345678
  /data_dir/year=2015/month=01/day=01/data02.12345678
  /data_dir/year=2015/month=01/day=01/data03.12345678
  </code></pre>
  
  <p>The 'directories' are listed as zero-byte files.</p>
  
  <p>I then run a spark step which needs to read these files. The loading code is thus:</p>
  
  <pre><code>sqlctx.read.json('hdfs:///data_dir, schema=schema)
  </code></pre>
  
  <p>The job fails with a java exception</p>
  
  <pre><code>java.io.IOException: Not a file: hdfs://10.159.123.38:9000/data_dir/year=2015
  </code></pre>
  
  <p>I had (perhaps naively) assumed that spark would recursively descend the 'dir tree' and load the data files. If I point to S3 it loads the data successfully.</p>
  
  <p>Am I misunderstanding HDFS? Can I tell spark to ignore zero-byte files? Can i use S3DistCp to flatten the tree?</p>