title: How to merge edits from one dataframe into another dataframe in Spark?
body: <p>I have a dataframe <strong>df1</strong> with 150 columns and many rows. I also have a dataframe <strong>df2</strong> with the same schema but very few rows containing edits that should be applied to <strong>df1</strong> (there's a key column <em>id</em> to identify which row to update). <strong>df2</strong> has only columns with updates populated. The other of the columns are null. What I want to do is to update the rows in <strong>df1</strong> with correspoding rows from dataframe <strong>df2</strong> in the following way:</p>
  
  <ul>
  <li>if a column in <strong>df2</strong> is null, it should not cause any changes in <strong>df1</strong></li>
  <li>if a column in <strong>df2</strong> contains a tilde "~", it should result in nullifying that column in <strong>df1</strong></li>
  <li>otherwise the value in column in <strong>df1</strong> should get replaced with the value from <strong>df2</strong></li>
  </ul>
  
  <p>How can I best do it? Can it be done in a generic way without listing all the columns but rather iterating over them? Can it be done using dataframe API or do I need to switch to RDDs?</p>
  
  <p>(Of course by updating dataframe df1 I mean creating a new, updated dataframe.)</p>
  
  <h2>Example</h2>
  
  <p>Let's say the schema is: id:Int, name:String, age: Int.</p>
  
  <p><strong>df1</strong> is:</p>
  
  <pre><code>1,"Greg",18
  2,"Kate",25
  3,"Chris",30
  </code></pre>
  
  <p><strong>df2</strong> is:</p>
  
  <pre><code>1,"Gregory",null
  2,~,26
  </code></pre>
  
  <p>The <strong>updated dataframe</strong> should look like this:</p>
  
  <pre><code>1,"Gregory",18
  2,null,26
  3,"Chris",30
  </code></pre>