title: Searching all file names recursively in hdfs using Spark
body: <p>I’ve been looking for a while now for a way to get all  <strong><em>filenames</em></strong> in a directory and its sub-directories in Hadoop file system (hdfs).</p>
  
  <p>I found out I can use these commands to get it :</p>
  
  <pre><code>sc.hadoopConfiguration.set("mapreduce.input.fileinputformat.input.dir.recursive", "true")
  sc.wholeTextFiles(path).map(_._1)
  </code></pre>
  
  <p>Here is "wholeTextFiles" documentation:</p>
  
  <blockquote>
    <p>Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file.</p>
    
    <p>Parameters:</p>
    
    <p>path - Directory to the input data files, the path can be
    comma separated paths as the list of inputs.</p>
    
    <p>minPartitions - A
    suggestion value of the minimal splitting number for input data.</p>
    
    <p>Returns: </p>
    
    <p>RDD representing tuples of file path and the corresponding
    file content</p>
    
    <p>Note: Small files are preferred, large file is also
    allowable, but may cause bad performance., On some filesystems,
    .../path/&#42; can be a more efficient way to read all files in a
    directory rather than .../path/ or .../path, Partitioning is
    determined by data locality. This may result in too few partitions by
    default.</p>
  </blockquote>
  
  <p>As you can see "wholeTextFiles" returns a pair RDD with both the filenames and their content. So I tried mapping it and taking only the file names, but I suspect it still reads the files.</p>
  
  <p>The reason I suspect so: if I try to count (for example) and I get the spark equivalent of "out of memory" (losing executors and not being able to complete the tasks).</p>
  
  <p>I would rather use Spark to achieve this goal the fastest way possible, however, if there are other ways with a reasonable performance I would be happy to give them a try.</p>
  
  <p><strong>EDIT:</strong>
  To clear it - I want to do it using Spark, I know I can do it using HDFS commands and such thing - I would like to know how to do such thing with the existing tools provided with Spark and maybe an explanation on how I can make "wholeTextFiles" not reading the text itself (kind of like how transformations only happen after an action  and some of the "commands" never really happen).</p>
  
  <p>Thank you very much!</p>