title: Converting data from .dat to parquet using Pyspark
body: <p>Why the number of rows is different after converting from .dat to parquet data format using pyspark? Even when I repeat the conversion on the same file multiple times, I get a different result (slightly more or slightly less or equal to the original rows count)!</p>
  
  <p>I am using my Macbook pro with 16 gb</p>
  
  <p>.dat file size is 16.5 gb</p>
  
  <p>spark-2.3.2-bin-hadoop2.7.</p>
  
  <p>I already have the rows count from my data provider (45 million rows).</p>
  
  <p>First I read the .dat file</p>
  
  <pre><code>2011_df = spark.read.text(filepath)
  </code></pre>
  
  <p>Second, I convert it to parquet, a process that takes about two hours.
      <code>2011_df.write.option("compression","snappy").mode("overwrite").save("2011.parquet")</code>  </p>
  
  <p>Afterwards, I read the converted parquet file</p>
  
  <pre><code>de_parq = spark.read.parquet(filepath)
  </code></pre>
  
  <p>Finally, I use "count" to get rows numbers.</p>
  
  <pre><code>de_parq.count()
  </code></pre>