title: S3 partition (file size) for effecient Athena query
body: <p>I have a pipeline that load daily records into S3. I then utilize AWS Glue Crawler to create partition for facilitating AWS Athena query. However, there is a large partitioned data, if compared to others.</p>
  
  <p>S3 folders/files are displayed as follows:</p>
  
  <pre><code>s3.ObjectSummary(bucket_name='bucket', key='database/table/2019/00/00/2019-00-00.parquet.gzip')   7.8 MB
  
  s3.ObjectSummary(bucket_name='bucket', key='database/table/2019/01/11/2019-01-11.parquet.gzip')  29.8 KB
  s3.ObjectSummary(bucket_name='bucket', key='database/table/2019/01/12/2019-01-12.parquet.gzip')  28.5 KB
  s3.ObjectSummary(bucket_name='bucket', key='database/table/2019/01/13/2019-01-13.parquet.gzip')  29.0 KB
  s3.ObjectSummary(bucket_name='bucket', key='database/table/2019/01/14/2019-01-14.parquet.gzip')  43.3 KB
  s3.ObjectSummary(bucket_name='bucket', key='database/table/2019/01/15/2019-01-15.parquet.gzip') 139.9 KB
  </code></pre>
  
  <p>with the file size displayed at the end of each line. Note that, <code>2019-00-00.parquet.gzip</code> contains all records before 2019-01-11 and therefore, its size is large. I have read <a href="https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/" rel="nofollow noreferrer">this</a> and it says that <em>"If your data is heavily skewed to one partition value, and most queries use that value, then the overhead may wipe out the initial benefit."</em></p>
  
  <p>So, I wonder should I split <code>2019-00-00.parquet.gzip</code> into smaller parquet files with different partitions. For example,</p>
  
  <pre><code>key='database/table/2019/00/00/2019-00-01.parquet.gzip',
  key='database/table/2019/00/00/2019-00-02.parquet.gzip',
  key='database/table/2019/00/00/2019-00-03.parquet.gzip', ......
  </code></pre>
  
  <p>However, I suppose this partitioning is not so useful as it does not reflect when were the old records stored. I am opened for all workarounds. Thank you.</p>