title: Spark remove duplicate rows from DataFrame
body: <p>Assume that I am having a DataFrame like : </p>
  
  <pre><code>val json = sc.parallelize(Seq("""{"a":1, "b":2, "c":22, "d":34}""","""{"a":3, "b":9, "c":22, "d":12}""","""{"a":1, "b":4, "c":23, "d":12}"""))
  val df = sqlContext.read.json(json)
  </code></pre>
  
  <p>I want to remove duplicate rows for column "a" based on the value of column "b". i.e, if there are duplicate rows for column "a", I want to keep the one with larger value for "b". For the above example, after processing, I need only </p>
  
  <blockquote>
    <p>{"a":3, "b":9, "c":22, "d":12}</p>
  </blockquote>
  
  <p>and </p>
  
  <blockquote>
    <p>{"a":1, "b":4, "c":23, "d":12}</p>
  </blockquote>
  
  <p>Spark DataFrame dropDuplicates API doesn't seem to support this. With the RDD approach, I can do a <code>map().reduceByKey()</code>, but what DataFrame specific operation is there to do this?</p>
  
  <p>Appreciate some help, thanks.</p>