title: Change filter/where condition when restarting a Structured Streaming query reading data from Delta Table
body: <p>In Structured Streaming, will the checkpoints keep track of which data has already been processed from a Delta Table?</p>
  <pre class="lang-py prettyprint-override"><code>def fetch_data_streaming(source_table: str):
    print(&quot;Fetching now&quot;)
    streamingInputDF = (
      spark
        .readStream
        .format(&quot;delta&quot;)
        .option(&quot;maxBytesPerTrigger&quot;,1024)
        .table(source_table)
        .where(&quot;measurementId IN (1351,1350)&quot;)
        .where(&quot;year &gt;= '2021'&quot;)
      )
  
    query = (
      streamingInputDF
        .writeStream
        .outputMode(&quot;append&quot;)
        .option(&quot;checkpointLocation&quot;, &quot;/streaming_checkpoints/5&quot;)
        .foreachBatch(customWriter)
        .start()
        .awaitTermination()
    )
    
  return query
  
  def customWriter(batchDF,batchId):
    print(batchId)
    print(batchDF.count())
    batchDF.show(10)
    length = batchDF.count()
    print(&quot;batchId,batch size:&quot;,batchId,length)
  </code></pre>
  <p>If I change the where clause in the streamingInputDF to add more measurentId, the structured streaming job doesn't always acknowledge the change and fetch the new data values. It continues to run as if nothing has changed, whereas at times it starts fetching new values.</p>
  <p>Isn't the checkpoint supposed to identify the change?</p>
  <p>Edit: Schema of delta table:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>col_name</th>
  <th>data_type</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>measurementId</td>
  <td>int</td>
  </tr>
  <tr>
  <td>year</td>
  <td>int</td>
  </tr>
  <tr>
  <td>time</td>
  <td>timestamp</td>
  </tr>
  <tr>
  <td>q</td>
  <td>smallint</td>
  </tr>
  <tr>
  <td>v</td>
  <td>string</td>
  </tr>
  </tbody>
  </table>
  </div>