title: PySpark: How to specify file size when writing parquet files?
body: <p>I have a 600+gb JSON that I want to convert to parquet. The syntax for reading and writing parquet is trivial:</p>
  <p>Reading:</p>
  <pre><code>data = spark.read.parquet('file-path')
  </code></pre>
  <p>Writing:</p>
  <pre><code>data.write.parquet(&quot;file-path&quot;)
  </code></pre>
  <p>My question, though, is whether there's an option to specify the size of the resultant parquet files, namely close to 128mb, which according to Spark's documetnation is the most performant size. Currently doing <code>data.write.parquet</code> results in files between 10-20mb, which I suspect is affecting the performance of my jobs.</p>
  <p>I've tried to use <code>coalesce</code>, as in <code>data.coalesce(1500).write.mode(&quot;append&quot;).parquet(&quot;file-path&quot;)</code> as a workaround, but setting the partition number is somewhat hit-or-miss.</p>
  <p>Is there a better way of writing to parquet?</p>