title: comparing two spark dataframe and generate new third dataframe with remark
body: <p>I have two spark dataframes, one has 70k rows and the other 60k. I am trying to compare each column string with other dataframe column and generate the new dataframe with Remark condition. If the column string from df1 is present in df2 it will be duplicated remark. I have tried using outer join but, it is not giving me the expected output.</p>
  
  <pre><code>df1
  colA    colB
  A       d4f488bef2
  B       c8a91953fc
  C       62026fd921
  D       e88480226d
  E       8335195031
  
  df2
  ColA    ColB
  W       411c78854c9
  X       0bfeb09d6cf
  C       62026fd9211
  E       8335195031e
  
  **Expected output : df3**
  ColA    ColB            Remark
  A       d4f488bef2d2    From df1
  B       c8a91953fc52    From df1
  D       e88480226d3b    From df1
  W       411c78854c9c    From df2
  X       0bfeb09d6cfb    From df2
  C       62026fd921      duplicate(In Both)
  E       8335195031      duplicate(In Both)
  
  </code></pre>