title: Eliminate duplicates (deduplication) in Streaming DataFrame
body: <p>I have a Spark streaming processor.
  The Dataframe dfNewExceptions has duplicates (duplicate by &quot;ExceptionId&quot;).
  Since this is a streaming dataset, the below query fails:</p>
  <pre><code>val dfNewUniqueExceptions = dfNewExceptions.sort(desc(&quot;LastUpdateTime&quot;))
                                      .coalesce(1)
                                      .dropDuplicates(&quot;ExceptionId&quot;)
                                      
  val dfNewExceptionCore = dfNewUniqueExceptions.select(&quot;ExceptionId&quot;, &quot;LastUpdateTime&quot;)
      dfNewExceptionCore.writeStream
        .format(&quot;console&quot;)
  //      .outputMode(&quot;complete&quot;)
        .option(&quot;truncate&quot;, &quot;false&quot;)
        .option(&quot;numRows&quot;,5000)
        .start()
        .awaitTermination(1000)
    
  </code></pre>
  <p>**
  Exception in thread &quot;main&quot; org.apache.spark.sql.AnalysisException: Sorting is not supported on streaming DataFrames/Datasets, unless it is on aggregated DataFrame/Dataset in Complete output mode;;
  **</p>
  <p>This is also documented here: <a href="https://home.apache.org/%7Epwendell/spark-nightly/spark-branch-2.0-docs/latest/structured-streaming-programming-guide.html" rel="nofollow noreferrer">https://home.apache.org/~pwendell/spark-nightly/spark-branch-2.0-docs/latest/structured-streaming-programming-guide.html</a></p>
  <p>Any suggestions on how the duplicates can be removed from dfNewExceptions?</p>