title: Spark: How to transform a Seq of RDD into a RDD
body: <p>I'm just starting in Spark &amp; Scala</p>
  
  <p>I have a directory with multiple files in it
  I successfully load them using </p>
  
  <pre><code>sc.wholeTextFiles(directory)
  </code></pre>
  
  <p>Now I want to go one level up. I actually have a directory that contains sub directories that contain files. My goal is to get an <code>RDD[(String,String)]</code> so I can move forward, where the <code>RDD</code> represents name and content of the file.</p>
  
  <p>I tried the following:</p>
  
  <pre><code>val listOfFolders = getListOfSubDirectories(rootFolder)
  val input = listOfFolders.map(directory =&gt; sc.wholeTextFiles(directory))
  </code></pre>
  
  <p>but I got a <code>Seq[RDD[(String,String)]]</code>
  How do I transform this <code>Seq</code> into an <code>RDD[(String,String)]</code> ?</p>
  
  <p>Or maybe I'm not doing things right and I should try a different approach?</p>
  
  <p><strong>Edit</strong>: added code</p>
  
  <pre><code>// HADOOP VERSION
  val rootFolderHDFS = "hdfs://****/"
  val hdfsURI = "hdfs://****/**/"
  
  // returns a list of folders (currently about 800)
  val listOfFoldersHDFS = ListDirectoryContents.list(hdfsURI,rootFolderHDFS)
  val inputHDFS = listOfFoldersHDFS.map(directory =&gt; sc.wholeTextFiles(directory))
  // RDD[(String,String)]
  //    val inputHDFS2 = inputHDFS.reduceRight((rdd1,rdd2) =&gt; rdd2 ++ rdd1)
  val init = sc.parallelize(Array[(String, String)]())
  val inputHDFS2 = inputHDFS.foldRight(init)((rdd1,rdd2) =&gt; rdd2 ++ rdd1)
  
  // returns org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError
  println(inputHDFS2.count)
  </code></pre>