title: Apache Spark groupByKey -> saveAsTextfile
body: <p>i have a huge XML File containing many unsorted rows:</p>
  
  <pre><code>&lt;row&gt;
    &lt;field name="f1"&gt;group&lt;/field&gt;
    &lt;field name="f2"&gt;number&lt;/field&gt;
    &lt;field name="f3"&gt;number2&lt;/field&gt;
  &lt;/row&gt;
  </code></pre>
  
  <p>and i want to use spark for extracting each group (f1) with distinct numbers into a seperate file.</p>
  
  <p>for now i tried to extract all groups (~ 50) first, using map, distinct and collect, then iterating over this group-array. using the original RDD, i filter all rows matching the current group, saveToTextFile.
   - works, but i am pretty sure that this way is the least efficient one.</p>
  
  <p>therefore i changed my xml importer to return (f1, (f2,f3)) and tried to:</p>
  
  <pre><code>rdd.groupByKey(numPartitions).mapPartitions(part =&gt; {
    part.map(data =&gt; {
      val group = data._1
      val numbers = data._2
      ...
  }
  </code></pre>
  
  <p>this is how far i get, because since numbers now is Iterable[(String,String)], and not an RDD.</p>
  
  <p>my idea was basicaly (pseudo code):</p>
  
  <pre><code> rdd.groupByKey(numPartitions).map((group, numbers) =&gt; { 
      numbers.distinct.map(OutputFormatter(_)).
        saveAsTextFile(s"$target$group") 
   }
  </code></pre>
  
  <p>numPartitions here should match the size of workers, to have one group per worker (i`d also like to benchmark behavior here, but this is OT)</p>
  
  <p>is it correct that numbers <em>cannot</em> be a RDD here, since groupByKey needs to collect data first?
  if not, what did i miss?</p>
  
  <p>is there a common best-practice for this operation? i am new to spark/hadoop/etc </p>
  
  <p>Thank you.</p>