title: How do I save a file in a Spark PairRDD using the key as the filename and the value as the contents?
body: <p>In Spark, I have downloaded multiple files from s3 using sc.binaryFiles. The RDD that results has the key as the filename and the value has the contents of the file. I have decompressed the file contents, csv parsed it, and converted it to a dataframe. So, now I have a PairRDD[String, DataFrame]. The problem I have is that I want to save the file to HDFS using the key as the filename and save the value as a parquet file overwriting one if it already exists. This is what I got so far.</p>
  
  <pre><code>val files = sc.binaryFiles(lFiles.mkString(","), 250).mapValues(stream =&gt; sc.parallelize(readZipStream(new ZipInputStream(stream.open))))
  val tables = files.mapValues(file =&gt; {
      val header = file.first.split(",")
      val schema = StructType(header.map(fieldName =&gt; StructField(fieldName, StringType, true)))
      val lines = file.mapPartitionsWithIndex { (idx, iter) =&gt; if (idx == 0) iter.drop(1) else iter }.flatMap(x =&gt; x.split("\n"))
      val rowRDD = lines.map(x =&gt; Row.fromSeq(x.split(",")))
      sqlContext.createDataFrame(rowRDD, schema)
  })
  </code></pre>
  
  <p>If you have any advice, please let me know. I would appreciate it.</p>
  
  <p>Thanks,
  Ben</p>