title: Comparing two data frames in Spark (performance)
body: <p>I need to compare two dataframes in my spark application. I went through the following post.
  <a href="https://stackoverflow.com/questions/36199901/how-to-obtain-the-difference-between-two-dataframes">How to obtain the difference between two DataFrames?</a></p>
  
  <p>However, I don't understand why the approach in the best answer</p>
  
  <pre><code>df1.unionAll(df2).except(df1.intersect(df2))
  </code></pre>
  
  <p>is better than the one in the question</p>
  
  <pre><code>df1.except(df2).union(df2.except(df1))
  </code></pre>
  
  <p>Can anyone explain?
  As per my understanding, the latter works with two smaller datasets and former works with a large dataset. Is it because the latter does a distinct as a part of union? Even then, if it is more likely case that two data frames have same records, we are dealing with a small dataset in the latter case. </p>