title: pytest assert for pyspark dataframe comparison
body: <p>I have 2 pyspark dataframe as shown in file attached. expected_df and actual_df</p>
  
  <p><a href="https://i.stack.imgur.com/nsVjx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nsVjx.png" alt="enter image description here"></a></p>
  
  <p>In my unit test I am trying to check if both are equal or not. </p>
  
  <p>for which my code is </p>
  
  <pre><code>expected = map(lambda row: row.asDict(), expected_df.collect()) 
  actual = map(lambda row: row.asDict(), actaual_df.collect()) 
  assert expected = actual 
  </code></pre>
  
  <p>Since both dfs are same but row order is different so assert fails here.
  What is best way to compare such dfs. </p>