title: How to efficiently select distinct rows on an RDD based on a subset of its columns`
body: <p>Consider a Case Class:</p>
  
  <pre><code>case class Prod(productId: String, date: String, qty: Int, many other attributes ..)
  </code></pre>
  
  <p>And an </p>
  
  <pre><code>val rdd: RDD[Prod]
  </code></pre>
  
  <p>containing many instances of that class.</p>
  
  <p>The unique key is intended to be the <code>(productId,date)</code> tuple. However we do have some duplicates.</p>
  
  <p>Is there any efficient means to remove the duplicates?</p>
  
  <p>The operation</p>
  
  <pre><code>      rdd.distinct
  </code></pre>
  
  <p>would look for <em>entire rows</em> that are duplicated.  </p>
  
  <p>A fallback would involve joining the unique <code>(productId,date)</code> combinations back to the entire rows: I am working through exactly how to do this.  But even so it is several operations. A simpler approach (faster as well?) would be useful if it exists.</p>