title: What is the reason for inconsistent counts in Pyspark, Spark SQL and toPandas().shape?
body: <p>I am working on databricks cloud 5.4 ML and I created a training dataset for my classification problem. When counting the records I get inconsistencies in the counts I cannot explain. </p>
  
  <p>Furthermore I have checked that my Spark DataFrame does not contain null or nan values. I also did the same check in Pandas.</p>
  
  <pre class="lang-py prettyprint-override"><code>#train is spark dataframe
  train.select([F.count(F.when(F.isnan(c) | F.col(c).isNull(),c)).alias(c) 
     for c in train.columns]).show() #all counts are 0
  train.filter((F.col('colname') == "") | F.col('colname').isNull() 
     | F.isnan(F.col('colname'))).count() #count is 0
  train.toPandas().isnull().sum().sum() #count is 0
  </code></pre>
  
  <p>Here under are the different count values:</p>
  
  <pre class="lang-py prettyprint-override"><code>train.count() #count is 6011
  train.toPandas().shape[0] #count is 6022
  </code></pre>
  
  <pre class="lang-sql prettyprint-override"><code>-- Spark SQL
  -- count is 6012
  SELECT COUNT(*)
  FROM train
  </code></pre>
  
  <pre class="lang-sql prettyprint-override"><code>-- Spark SQL
  -- count is 5985
  SELECT COUNT(colname)
  FROM train
  </code></pre>
  
  <p>I would expect that the record count generated by Pyspark, Spark SQL and toPandas().shape to be the same. What are your thoughts? Solutions?</p>
  
  <p>Thanks a lot.</p>