title: How to join two data frames in Apache Spark and merge keys into one column?
body: <p>I have two following Spark data frames:</p>
  
  <pre><code>sale_df:
  
  |user_id|total_sale|
  +-------+----------+
  |      a|      1100|
  |      b|      2100|
  |      c|      3300|
  |      d|      4400  
  </code></pre>
  
  <p>and target_df:</p>
  
  <pre><code> user_id|personalized_target|
  +-------+-------------------+
  |      b|               1000|
  |      c|               2000|
  |      d|               3000|
  |      e|               4000|
  +-------+-------------------+
  </code></pre>
  
  <p>How can I join them in a way that output is:</p>
  
  <pre><code>user_id   total_sale   personalized_target
   a           1100            NA
   b           2100            1000
   c           3300            2000
   d           4400            4000
   e           NA              4000
  </code></pre>
  
  <p>I have tried all most all the join types but it seems that single join can not make the desired output.</p>
  
  <p>Any PySpark or SQL and HiveContext can help.</p>