title: Writing to hadoop distributed file system multiple times with Spark
body: <p>I've created a spark job that reads in a textfile everyday from my hdfs and extracts unique keys from each line in the text file. There are roughly 50000 keys in each text file. The same data is then filtered by the extracted key and saved to the hdfs.</p>
  
  <p>I want to create a directory in my hdfs with the structure: hdfs://.../date/key that contains the filtered data. The problem is that writing to the hdfs takes a <strong>very very</strong> long time because there are so many keys.</p>
  
  <p>The way it's written right now:</p>
  
  <pre><code>val inputData = sparkContext.textFile(""hdfs://...", 2)
  val keys = extractKey(inputData) //keys is an array of approx 50000 unique strings
  val cleanedData = cleanData(inputData) //cleaned data is an RDD of strings
  keys.map(key =&gt; {
      val filteredData = cleanedData.filter(line =&gt; line.contains(key))
      filteredData.repartition(1).saveAsTextFile("hdfs://.../date/key")
  })
  </code></pre>
  
  <p>Is there a way to make this faster? I've thought about repartitioning the data into the number of keys extracted but then I can't save in the format hdfs://.../date/key. I've also tried groupByKey but I can't save the values because they aren't RDDs.</p>
  
  <p>Any help is appreciated :)</p>