title: Increasing Spark Read and Parquet Conversion Performance for Gzipped Text File
body: <p>Use case: 
  A> Have Text Gzipped files in AWS s3 location
  B> Hive Table created on top of the file, to access the data from the file as Table
  C> Using Spark Dataframe to read the table and converting into Parquet Data with Snappy Compression
  D> Number of fields in the table is 25, which includes 2 partition columns. Data Type is String except for two fields which has Decimal as data type.</p>
  
  <p>Used following Spark Option: --executor-memory 37G --executor-cores 5 --num-executors 20</p>
  
  <p>Cluster Size - 10 Data Nodes of type r3.8xLarge</p>
  
  <p>Found the number of vCores used in AWS EMR is always equal to the number of files, may be because gzip files are not splittable. Gzipped files are coming from different system and size of files are  around 8 GB.</p>
  
  <p>Total Time taken is more than 2 hours for Parquet conversion for 6 files with total size 29.8GB. </p>
  
  <p>Is there a way to improve the performance via Spark, using version 2.0.2?</p>
  
  <p>Code Snippet:</p>
  
  <p>val srcDF = spark.sql(stgQuery)
  srcDF.write.partitionBy("data_date","batch_number").options(Map("compression"->"snappy","spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version"->"2","spark.speculation"->"false")).mode(SaveMode.Overwrite).parquet(finalPath)</p>