title: Apache SPARK GroupByKey alternate
body: <p>I have below columns in my table [col1,col2,key1,col3,txn_id,dw_last_updated]. Out of these txn_id , key1 are primary key columns. In my dataset I can have multiple records for the combination of (txn_id,key). Out of those records , I need to pick the latest one one based on dw_last_updated..</p>
  
  <p>I'm using a logic this. I'm consistently hitting memory issue and I believe its partly because of groupByKey()... Is there a better alternate for this ?</p>
  
  <pre><code>case class Fact(col1: Int,
    col2: Int,
    key1: String,
    col3: Int,
    txn_id: Double,
    dw_last_updated: Long)
  
  sc.textFile(s3path).map { row =&gt;
            val parts = row.split("\t")
            Fact(parts(0).toInt,
              parts(1).toInt,
              parts(2),
              parts(3).toInt,
              parts(4).toDouble,
              parts(5).toLong)
          }).map { t =&gt; ((t.txn_id, t.key1), t) }.groupByKey(512).map {
            case ((txn_id, key1), sequence) =&gt;
              val newrecord = sequence.maxBy {
                case Fact_Cp(col1, col2, key1, col3, txn_id, dw_last_updated) =&gt; dw_last_updated.toLong
              }
             (newrecord.col1 + "\t" + newrecord.col2 + "\t" + newrecord.key1 +
                "\t" + newrecord.col3 + "\t" + newrecord.txn_id + "\t" + newrecord.dw_last_updated)
          }
  </code></pre>
  
  <p>Appreciate your thoughts / suggestions...</p>