title: DropDuplicates is not giving expected result
body: <p>I am working on a use-case of removing duplicate records from incoming structured data (in the form of CSV files within a folder on HDFS). In order to try this use-case, I wrote some sample code using files option to see if duplicates can be removed from the records that are present in the CSVs that are copied to the folder (HDFS). </p>
  
  <p>Find below the codepiece: </p>
  
  <pre class="lang-scala prettyprint-override"><code>import org.apache.spark.sql.functions._
  import org.apache.spark.sql.SparkSession
  import spark.implicits._
  import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType}
  
  
  val spark = SparkSession.builder.appName("StructuredNetworkWordCount").getOrCreate()
  val userSchema = new StructType()
      .add("prod_code", "string")
      .add("bal", "integer")
      .add("v_txn_id", "string")
      .add("timestamp", "Timestamp")
  
  val csvDF = spark.readStream.option("sep", ",")
                              .schema(userSchema)
                              .csv("/user/Temp")
  csvDF.dropDuplicates("v_txn_id")
  csvDF.createOrReplaceTempView("table1")
  
  val dbDf2 = spark.sql("select prod_code, bal, v_txn_id, current_timestamp timestamp from table1")
  dbDf2.writeStream.queryName("aggregates").outputMode("update").format("memory").start()
  
  spark.sql("select * from aggregates").show();
  </code></pre>
  
  <p>Now, when I copy a file in the folder with duplicate records (by v_txn_id), i still see that the result sink gets all the rows from the file: </p>
  
  <pre><code>P1,1000,TXNID1
  P1,2000,TXNID2
  P1,3000,TXNID2
  P1,4000,TXNID3
  P1,5000,TXNID3
  P1,6000,TXNID4
  </code></pre>
  
  <p>All these rows in the csv file get moved to the result "aggregates". What I am expecting is: </p>
  
  <pre><code>P1,1000,TXNID1
  P1,3000,TXNID2
  P1,5000,TXNID3
  P1,6000,TXNID4
  </code></pre>
  
  <p>This is the first time I am attempting structured streaming (with state), so pardon me for trivial question. Any suggestions would help a lot. </p>