title: Spark EMR Read from S3 Transform and Write back to S3, Need Performance Improvement
body: <p><strong>Using AWS EMR cluster.</strong></p>
  <p><strong>Scenario :</strong></p>
  <ol>
  <li>I need to read inpute parquet files from S3</li>
  <li>Do some transformation on data, I do it by Spark SQL</li>
  <li>I need to write output files(multiplart parquet) back to S3.</li>
  </ol>
  <ul>
  <li>Input data volume : 30GB</li>
  <li>Output volume : 500GB</li>
  </ul>
  <p>I am using 1 Master - m5.8xlarge and 7 Core nodes of type r5.12xlarge.</p>
  <p>I tried writing files direct to S3 in step3 as well as writing to local files and move them to S3.
  Writing direct to S3 taking more time.</p>
  <ul>
  <li>Time writing direct to S3 = 5 hr</li>
  <li>Time writing to local and move to S3 = 4 hr</li>
  </ul>
  <ol>
  <li>I tried with m4.2xlarge Master with EBS of 600GB and 10 Core r5.2xlarge, and tried writing files to /mnt/ path. But, Cluster core nodes fails due to space issue.
  Is there any way, that I can ask Core nodes to write files on Master node's EBS volume ?</li>
  </ol>
  <p>As the files total size in Step3 are &gt;500GB, I have to use r5.12xlarge instances. For lower instances, it gives out-of-memory issue.</p>
  <p>I am not doing repartition on dataFrame to avoid data suffling. Just doing dataFrame.write.parquet(path)</p>
  <ol start="2">
  <li>Any other Master/Core/EBS combination for optimal performance  ?</li>
  </ol>
  <p>Please suggest best combination for my scenario.</p>