title: Scala & DataBricks: Getting a list of Files
body: <p>I am trying to make a list of files in an S3 bucket on Databricks within Scala, and then split by regex.  I am very new to Scala. The python equivalent would be </p>
  
  <pre><code>all_files = map(lambda x: x.path, dbutils.fs.ls(folder))
  filtered_files = filter(lambda name: True if pattern.match(name) else False, all_files)
  </code></pre>
  
  <p>but I want to do this in Scala.</p>
  
  <p>From <a href="https://alvinalexander.com/scala/how-to-list-files-in-directory-filter-names-scala" rel="nofollow noreferrer">https://alvinalexander.com/scala/how-to-list-files-in-directory-filter-names-scala</a></p>
  
  <pre><code>import java.io.File
  def getListOfFiles(dir: String):List[File] = {
      val d = new File(dir)
      if (d.exists &amp;&amp; d.isDirectory) {
          d.listFiles.filter(_.isFile).toList
      } else {
          List[File]()
      }
  }
  </code></pre>
  
  <p>However, this produces an empty list.</p>
  
  <p>I've also thought of </p>
  
  <pre><code>var all_files: List[Any] = List(dbutils.fs.ls("s3://bucket"))
  </code></pre>
  
  <p>but this produces a list of things like (with length 1)</p>
  
  <pre><code>all_files: List[Any] = List(WrappedArray(FileInfo(s3://bucket/.internal_name.pl.swp, .internal_name.pl.swp, 12288), FileInfo(s3://bucket/file0, 10223616), FileInfo(s3://bucket/, file1, 0), ....)
  </code></pre>
  
  <p>which has a length of 1.  I cannot turn this into a dataframe, as suggested by <a href="https://stackoverflow.com/questions/38257630/how-to-iterate-scala-wrappedarray-spark">How to iterate scala wrappedArray? (Spark)</a> This isn't usable.</p>
  
  <p>How can I generate a list of files in Scala, and then iterate through them?</p>