title: Update only changed rows pyspark delta table databricks
body: <p>There is a requirement to update only changed rows in an existing table compared to the created dataframe. So rightnow , i do subtract and get the changed rows, but not sure how to merge into existing table.</p>
  <pre><code>old_df = spark.sql(&quot;select * from existing table&quot;)
  diff = new_df.subtract(old_df)
  </code></pre>
  <p>diff dataframe has to be now inserted(if new rows) or update existing records</p>
  <pre><code>(deltaTable.alias(&quot;full_df&quot;).merge(
      merge_df.alias(&quot;append_df&quot;),
      &quot;full_df.col1 = append_df.col1 OR full_df.col2 =append_df.col2&quot;) 
    .whenNotMatchedInsertAll() 
    .execute()
  )
  </code></pre>
  <p>This is not updating existing records (case: col2 value changed ; col1 not changed)</p>