title: How to drop duplicates considering only subset of columns?
body: <p>I use Spark 1.6 and am doing inner join on two dataframes as follows:</p>
  
  <pre><code>val filtergroup = metric
    .join(filtercndtns, Seq("aggrgn_filter_group_id"), inner)
    .distinct()
  </code></pre>
  
  <p>But I keep getting duplicate values in <code>aggrgn_filter_group_id</code> column. Can you please suggest some solution?</p>