title: DataFrame equality in Apache Spark
body: <p>Assume <code>df1</code> and <code>df2</code> are two <code>DataFrame</code>s in Apache Spark, computed using two different mechanisms, e.g., Spark SQL vs. the Scala/Java/Python API. </p>
  
  <p>Is there an idiomatic way to determine whether the two data frames are equivalent (equal, isomorphic), where equivalence is determined by the data (column names and column values for each row) being identical save for the ordering of rows &amp; columns?</p>
  
  <p>The motivation for the question is that there are often many ways to compute some big data result, each with its own trade-offs. As one explores these trade-offs, it is important to maintain correctness and hence the need to check for the equivalence/equality on a meaningful test data set.</p>