title: The pyspark groupby generates multiple rows in output with String groupby key
body: <p>The pyspark groupby operation does not produce unique group keys for large data sets</p>
  
  <p>I see repeated keys in the final output. </p>
  
  <pre><code> new_df = df.select('key','value') \
          .where(...) \
          .groupBy('key') \
          .count() 
  
  </code></pre>
  
  <p>e.g. above query returns multiple rows for a groupBy column (key). The datatype for groupby column('key') is string.</p>
  
  <p>I'm storing output in CSV by doing</p>
  
  <pre><code>new_df.write.format("csv") \
          .option("header", "true") \
          .mode("Overwrite") \
          .save(CSV_LOCAL_PATH)
  </code></pre>
  
  <p>e.g. output in CSV has duplicate rows</p>
  
  <pre><code>key1, 10
  key2, 20
  key1, 05
  </code></pre>
  
  <p>Tested in Spark 2.4.3 and 2.3
  There are duplicates. There is no difference in keys. This happens for multiple KEYS.</p>
  
  <p>It gives 1, when I count the rows for particular keys.</p>
  
  <pre><code>new_df.select('key','total')\
   .where((col("key") == "key1"))\
   .count()
  </code></pre>
  
  <p>I'm not sure if pyarrow settings make any difference. I'd it enabled before. I tried with both enabling and disabling pyarrow but the same result.</p>
  
  <p>spark.conf.set("spark.sql.execution.arrow.enabled", "true")</p>