title: Rolling your own reduceByKey in Spark Dataset
body: <p>I'm trying to learn to use DataFrames and DataSets more in addition to RDDs.  For an RDD, I know I can do <code>someRDD.reduceByKey((x,y) =&gt; x + y)</code>, but I don't see that function for Dataset.  So I decided to write one. </p>
  
  <pre><code>someRdd.map(x =&gt; ((x.fromId,x.toId),1)).map(x =&gt; collection.mutable.Map(x)).reduce((x,y) =&gt; {
    val result = mutable.HashMap.empty[(Long,Long),Int]
    val keys = mutable.HashSet.empty[(Long,Long)]
    y.keys.foreach(z =&gt; keys += z)
    x.keys.foreach(z =&gt; keys += z)
    for (elem &lt;- keys) {
      val s1 = if(x.contains(elem)) x(elem) else 0
      val s2 = if(y.contains(elem)) y(elem) else 0
      result(elem) = s1 + s2
    }
    result
  })
  </code></pre>
  
  <p>However, this returns everything to the driver.  How would you write this to return a <code>Dataset</code>?  Maybe mapPartition and do it there?</p>
  
  <p>Note this compiles but does not run because it doesn't have encoders for <code>Map</code> yet</p>