title: How to optimize groupBy() operation on Spark RDD
body: <p>My question is about alternatives/optimization to <code>groupBy()</code> operation on RDD. I have millions of <code>Message</code> instances which needs to be grouped based on some ID. I use <code>groupBy()</code> to accomplish the goal but I know the fact that it is  an expensive one and it takes huge amount of processing time .</p>
  
  <p>So I tried <code>reduceByKey(func)</code> and <code>combineByKey()</code> as an alternative which was suggested in some of resources that I went through but its not a fit to my scenario where 
   <strong><em>My requirement is just grouping</em></strong>. But my data size is huge where <code>groupBy</code> consumes more <strong>than 70% of my job's time</strong>.So any suggestions or solution are appreciated.  </p>
  
  <p><strong>My input Data</strong></p>
  
  <blockquote>
    <p>Seq(new Message("1", "abc", "Request"), new Message("1", "cba",
    "Response"), new Message("2", "def", "Request"), new Message("2",
    "fed", "Response"), new Message("3", "ghi", "Request"), new
    Message("3", "ihg", "Response"))</p>
  </blockquote>
  
  <p>For the sake of simplicity, i have given input data in <code>Seq</code> </p>
  
  <pre><code> def groupMessages(sourceRdd: RDD[Message]): Unit {
  
      val messageIdRdd = sourceRdd.mapPartitions(partitionData =&gt; {
        partitionData.map(row =&gt; (row.uniqueID(), row))
  
      })
  
      /*
       Group messages based on its message id. This functionality is to combine request response
       together as single transaction based on message id.
       */
      val groupbyRdd = messageIdRdd.groupBy(_._1) 
  }
  </code></pre>
  
  <p><strong>Expected Result</strong> </p>
  
  <blockquote>
    <p>(2,CompactBuffer((2,Message(2,def,Request)),
    (2,Message(2,fed,Response))))
    (3,CompactBuffer((3,Message(3,ghi,Request)),
    (3,Message(3,ihg,Response))))
    (1,CompactBuffer((1,Message(1,abc,Request)),
    (1,Message(1,cba,Response))))</p>
  </blockquote>