title: Writing large spark data frame as parquet to s3 bucket
body: <p><strong>My Scenario</strong></p>
  <ul>
  <li>I have a spark data frame in a AWS glue job with 4 million records</li>
  <li>I need to write it as a SINGLE parquet file in AWS s3</li>
  </ul>
  <p><strong>Current code</strong></p>
  <pre><code>file_spark_df.write.parquet(&quot;s3://&quot;+target_bucket_name)
  </code></pre>
  <p><strong>Issue</strong>
  the above code creates 100+ files each 17.8 to 18.1 MB in size , guess its some default break down size</p>
  <p><strong>Ques 1 : How do I create just one file ? for one spark data frame ?</strong>
  I checked <a href="https://spark.apache.org/docs/latest/sql-data-sources-parquet.html" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/sql-data-sources-parquet.html</a> didnt find any parameter to set</p>
  <p><strong>Ques 2 : How do I specify the name of the file</strong>
  I tried ...</p>
  <pre><code>file_df.write.parquet(&quot;s3://&quot;+target_bucket_name+&quot;/&quot;+target_file_name)
  </code></pre>
  <p>It created 100+ files inside &quot;s3://&quot;+target_bucket_name+&quot;/&quot;+target_file_name</p>
  <p><strong>Ques 3 : How do I specify the name of the file</strong>
  I need to create sub folders inside base3 bucket following code can do the job</p>
  <pre><code>file_df.write.parquet(&quot;s3://&quot;+target_bucket_name+&quot;/&quot;+today_date+&quot;/&quot;+target_file_name)
  </code></pre>
  <p>not sure if its the best way ... or there is a better way ?</p>