title: Spark 2.x - gzip vs snappy compression for parquet files
body: <p>I am (for the first time) trying to repartition the data my team is working with to enhance our querying performance. Our data is currently stored in partitioned .parquet files compressed with gzip. I have been reading that using snappy instead would significantly increase throughput (we query this data daily for our analysis). I still wanted to benchmark the two codecs to see the perfomance gap with with my own eyes. I wrote a simple (Py)Spark 2.1.1 application to carry out some tests. I persisted 50 millions records in memory (deserialized) in a single partition, wrote them into a single parquet file (to HDFS) using the different codecs and then imported the files again to assess the difference. My problem is that I can't see any significant difference for both read and write.</p>
  
  <p>Here is how I wrote my records to HDFS (same thing for the gzip file, just replace 'snappy' with 'gzip') :</p>
  
  <pre><code>persisted_records.write\
  .option('compression', 'snappy')\
  .mode("overwrite")\
  .partitionBy(*partition_cols)\
  .parquet('path_to_dir/test_file_snappy')
  </code></pre>
  
  <p>And here is how I read my single .parquet file (same thing for the gzip file, just replace 'snappy' with 'gzip') :</p>
  
  <pre><code>df_read_snappy = spark.read\
  .option('basePath', 'path_to_dir/test_file_snappy')\
  .option('compression', 'snappy')\
  .parquet('path_to_dir/test_file_snappy')\
  .cache()
  
  df_read_snappy.count()
  </code></pre>
  
  <p>I looked at the durations in the Spark UI. For information, the persisted (deserialized) 50 millions rows amount 317.4M. Once written into a single parquet file, the file weights 60.5M and 105.1M using gzip and snappy respectively (this is expected as gzip is supposed to have a better compression ratio). Spark spends 1.7min (gzip) et 1.5min (snappy) to write the file (single partition so a single core has to carry out all the work). Reading times amount to 2.7min (gzip) et 2.9min (snappy) on a single core (since we have a single file / HDFS block). This what I do not understand : where is snappy's higher performance ?</p>
  
  <p>Have I done something wrong ? Is my "benchmarking protocol" flawed ? Is the performance gain here but I am not looking at the right metrics ?</p>
  
  <p>I must add that I am using Spark default conf. I did not change anything aside from specifying the number of executors, etc. </p>
  
  <p>Many thanks for your help!</p>
  
  <p>Notice: Spark parquet jar version is 1.8.1</p>