title: Spark Scala list folders in directory
body: <p>I want to list all folders within a hdfs directory using Scala/Spark.
  In Hadoop I can do this by using the command: <code>hadoop fs -ls hdfs://sandbox.hortonworks.com/demo/</code></p>
  
  <p>I tried it with:</p>
  
  <pre><code>val conf = new Configuration()
  val fs = FileSystem.get(new URI("hdfs://sandbox.hortonworks.com/"), conf)
  
  val path = new Path("hdfs://sandbox.hortonworks.com/demo/")
  
  val files = fs.listFiles(path, false)
  </code></pre>
  
  <p>But it does not seem that he looks in the Hadoop directory as i cannot find my folders/files.</p>
  
  <p>I also tried with:</p>
  
  <pre><code>FileSystem.get(sc.hadoopConfiguration).listFiles(new Path("hdfs://sandbox.hortonworks.com/demo/"), true)
  </code></pre>
  
  <p>But this also does not help.</p>
  
  <p>Do you have any other idea?</p>
  
  <p>PS: I also checked this thread: <a href="https://stackoverflow.com/questions/27023766/spark-iterate-hdfs-directory">Spark iterate HDFS directory</a> but it does not work for me as it does not seem to search on hdfs directory, instead only on the local file system with schema file//.</p>