title: spark how to merge two dataframe on several columns?
body: <p>I have two dataframe </p>
  
  <ul>
  <li>a columns: ['q1', 'q2', 'q3', 'a1', 'a2']</li>
  <li>b columns: ['q1', 'q2', 'q3', 'b', 'b2']</li>
  </ul>
  
  <p>a may have some ['q1', 'q2', 'q3'] not in b , b also may have some ['q1', 'q2', 'q3']  not in a .</p>
  
  <p>Merge means, if a have ['q1', 'q2', 'q3'] same as b, join the rows , then union left rows . <code>full outer join</code> doesn't do such thing.</p>
  
  <p>my presudo code is like :</p>
  
  <pre><code>c = a.join(b, on= ['q1', 'q2', 'q3'], how='inner')
  c = c.union(a.filter( ~a.withColumn('xxx', F.concat_ws('|', 'q1', 'q2', 'q3') ).isin(c.select(F.concat_ws('|', 'q1', 'q2', 'q3')) )
  c = c.union(b.filter( ~b.withColumn('xxx', F.concat_ws('|', 'q1', 'q2', 'q3') ).isin(c.select(F.concat_ws('|', 'q1', 'q2', 'q3')) )
  </code></pre>
  
  <p>But this is very unefficient.</p>
  
  <p>Is there any better way ?</p>