title: PySpark Structured Streaming - Watermark
body: <p>I am learning Spark Structured Streaming using Python from Spark website. While implementing Watermark concept on streaming data, i am not getting expected output.
  In Batch 2, I am not supposed to get rows 2 &amp; 7 and should get only row 7 as i am using update mode. Can someone clarify why this happened?</p>
  
  <p>Here is the code:-</p>
  
  <pre><code># Start Streaming
  from pyspark.sql.functions import window
  
  streamingData = spark\
              .readStream\
              .schema(fileDF.schema)\
              .option('maxFilesPerTrigger', 1)\
              .option('header', True)\
              .format('com.databricks.spark.csv')\                  
  .load('/PySpark/Files/Source/Streaming/watermark*.csv')
  
  windowedCounts = streamingData \
              .withWatermark("timestamp", "10 minutes") \
              .groupBy(
                  window(streamingData.timestamp, "10 minutes", "5 minutes"),
                  streamingData.name) \
              .count()
  
  # Write Stream
  windowedCounts\
   .writeStream\
   .format('memory')\
   .queryName('watermark_stream')\
   .outputMode('update')\
   .start()
  </code></pre>
  
  <p><strong>Run 1:</strong></p>
  
  <p>Input:</p>
  
  <p>timestamp,name</p>
  
  <p>2018-07-19 13:27:00,dog</p>
  
  <p>2018-07-19 13:28:00,owl</p>
  
  <p>Output:</p>
  
  <p>0  (2018-07-19 13:20:00, 2018-07-19 13:30:00)  dog      1</p>
  
  <p>1  (2018-07-19 13:20:00, 2018-07-19 13:30:00)  owl      1</p>
  
  <p>2  (2018-07-19 13:25:00, 2018-07-19 13:35:00)  dog      1</p>
  
  <p>3  (2018-07-19 13:25:00, 2018-07-19 13:35:00)  owl      1</p>
  
  <p><strong>Run 2:</strong></p>
  
  <p>Input:</p>
  
  <p>2018-07-19 13:34:00,dog</p>
  
  <p>2018-07-19 13:29:00,cat</p>
  
  <p>Output:</p>
  
  <p>0  (2018-07-19 13:20:00, 2018-07-19 13:30:00)  dog      1</p>
  
  <p>1  (2018-07-19 13:20:00, 2018-07-19 13:30:00)  owl      1</p>
  
  <p>2  (2018-07-19 13:25:00, 2018-07-19 13:35:00)  dog      1</p>
  
  <p>3  (2018-07-19 13:25:00, 2018-07-19 13:35:00)  owl      1</p>
  
  <p>4  (2018-07-19 13:30:00, 2018-07-19 13:40:00)  dog      1</p>
  
  <p>5  (2018-07-19 13:25:00, 2018-07-19 13:35:00)  cat      1</p>
  
  <p>6  (2018-07-19 13:20:00, 2018-07-19 13:30:00)  cat      1</p>
  
  <p>7  (2018-07-19 13:25:00, 2018-07-19 13:35:00)  dog      2</p>