title: Pyspark collect() method gives different order when executing tests in Pycharm or Console
body: <p>I have some tests on my pytest suite that compare dataframes with <code>assert df1.collect() == df2.collect()</code>.</p>
  <p>If I execute the code inside the Pychar IDE the tests passes, if I execute the tests in console an assertion error is raised.</p>
  <p>After some debugging, I found that when I execute the test with the console the collected results are disordered.</p>
  <p>For example, if my dataframe has two rows, this code will pass in Pycharm but it fails in console:</p>
  <pre><code> assert df1.collect()[0] == df2.collect()[0]
  </code></pre>
  <p>And this one will fail in Pycharm but it will pass in console:</p>
  <pre><code>assert df1.collect()[1] == df2.collect()[0]
  </code></pre>
  <p>I've tried to invoke pytest with <code>python3 -m pytest</code> and just with <code>pytest</code>. Pycharm and the console are using the same venv</p>