title: Save the parquet output file with fixed size in spark
body: <p>I have 160GB of data,partition on <code>DATE</code> Column and storing in parquet file format running on spark 1.6.0.
  I need to store the output parquet files with equal sized files in each partition with fixed size say like 100MB each.</p>
  <p>I tried with below code:</p>
  <pre class="lang-scala prettyprint-override"><code>val blockSize= 1024*1024*100
  sc.hadoopConfiguration.setInt(&quot;dfs.blocksize&quot;, blockSize)
  sc.hadoopConfiguration.setInt(&quot;parquet.block.size&quot;,blockSize)
  
  df1.write.partitionBy(&quot;DATE&quot;).parquet(&quot;output_file_path&quot;)
  </code></pre>
  <p>The above configuration is not working, it is creating multiple files with default number of partitions,not the 100 MB file.</p>