title: How to update a pyspark dataframe with new values from another dataframe?
body: <p>I have two spark dataframes:</p>
  
  <p>Dataframe A:</p>
  
  <pre><code>|col_1 | col_2 | ... | col_n |
  |val_1 | val_2 | ... | val_n |
  </code></pre>
  
  <p>and dataframe B:</p>
  
  <pre><code>|col_1 | col_2 | ... | col_m |
  |val_1 | val_2 | ... | val_m |
  </code></pre>
  
  <p>Dataframe B can contain duplicate, updated and new rows from dataframe A. I want to write an operation in spark where I can create a new dataframe containing the rows from dataframe A and the updated and new rows from dataframe B.</p>
  
  <p>I started by creating a hash column containing only the columns that are not updatable. This is the unique id. So let's say <code>col1</code> and <code>col2</code> can change value (can be updated), but <code>col3,..,coln</code> are unique. I have created a hash function as <code>hash(col3,..,coln)</code>:</p>
  
  <pre><code>A=A.withColumn("hash", hash(*[col(colname) for colname in unique_cols_A]))
  B=B.withColumn("hash", hash(*[col(colname) for colname in unique_cols_B]))
  </code></pre>
  
  <p>Now I want to write some spark code that basically selects the rows from B that have the hash not in A <strong>(so new rows and updated rows)</strong> and join them into a new dataframe together with the rows from A. How can I achieve this in pyspark?</p>
  
  <p>Edit:
  Dataframe B can have extra columns from dataframe A, so a union is not possible.</p>
  
  <p>Sample example</p>
  
  <p>Dataframe A:</p>
  
  <pre><code>+-----+-----+
  |col_1|col_2|
  +-----+-----+
  |    a|  www|
  |    b|  eee|
  |    c|  rrr|
  +-----+-----+
  </code></pre>
  
  <p>Dataframe B:</p>
  
  <pre><code>+-----+-----+-----+
  |col_1|col_2|col_3|
  +-----+-----+-----+
  |    a|  wew|    1|
  |    d|  yyy|    2|
  |    c|  rer|    3|
  +-----+-----+-----+
  </code></pre>
  
  <p>Result:
  Dataframe C:</p>
  
  <pre><code>+-----+-----+-----+
  |col_1|col_2|col_3|
  +-----+-----+-----+
  |    a|  wew|    1|
  |    b|  eee| null|
  |    c|  rer|    3|
  |    d|  yyy|    2|
  +-----+-----+-----+
  </code></pre>