title: Recursively fetch file contents from subdirectories using sc.textFile
body: <p>It seems that SparkContext textFile expects only files to be present in the given directory location - it does not either </p>
  
  <ul>
  <li>(a) recurse or</li>
  <li>(b) even <em>support</em> directories (tries to read directories as files)</li>
  </ul>
  
  <p>Any suggestion how to structure a recursion - potentially simpler than creating the recursive file list / descent logic manually?</p>
  
  <p>Here is the use case: files under </p>
  
  <blockquote>
    <p>/data/tables/my_table</p>
  </blockquote>
  
  <p>I want to be able to read via an hdfs call all the files at all directory levels under that parent directory.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>The sc.textFile() invokes the Hadoop FileInputFormat via the (subclass) TextInputFormat.  Inside the logic does exist to do the recursive directory reading - i.e. first detecting if an entry were a directory and if so then descending:</p>
  
  <pre><code>&lt;!-- language: java --&gt;
       for (FileStatus globStat: matches) {
  218          if (globStat.isDir()) {
  219            for(FileStatus stat: fs.listStatus(globStat.getPath(),
  220                inputFilter)) {
  221              result.add(stat);
  222            }          
  223          } else {
  224            result.add(globStat);
  225          }
  226        }
  </code></pre>
  
  <p>However when invoking sc.textFile there are errors on directory entries: "not a file". This behavior is confusing - given the proper support appears to be in place for handling directories.</p>