title: How debug spark dropduplicate and join function calls?
body: <p>There is some table with duplicated rows. I am trying to reduce duplicates and stay with latest <code>my_date</code> (if there are
  rows with same <code>my_date</code> it is no matter which one to use)</p>
  
  <pre><code>val dataFrame = readCsv()
    .dropDuplicates("my_id", "my_date")
    .withColumn("my_date_int", $"my_date".cast("bigint"))
  
  import org.apache.spark.sql.functions.{min, max, grouping}
  
  val aggregated = dataFrame
      .groupBy(dataFrame("my_id").alias("g_my_id"))
    .agg(max(dataFrame("my_date_int")).alias("g_my_date_int"))
  
  val output = dataFrame.join(aggregated, dataFrame("my_id") === aggregated("g_my_id") &amp;&amp; dataFrame("my_date_int") === aggregated("g_my_date_int"))
      .drop("g_my_id", "g_my_date_int")
  </code></pre>
  
  <p>But after this code I when grab <code>distinct my_id</code> I get about <code>3000</code> less than in source table. What a reason can be?</p>
  
  <p>how to debug this situation? </p>