title: How to get in-memory size of the ParquetWriter object before closing it and writing it to disk?
body: <p>I am reading Avro messages from a stream and writing them out into a Parquet file using parquet.hadoop.ParquetWriter. I am trying to have the size of the output files above a threshold limit. the problem is that ParquetWriter keeps everything in memory and only writes it out to disk at the end when writer is closed. Based on Parquet documentation, the data is written into memory object in the final format, meaning that the size of the object in memory is the same as the final size on disk. My question is how to get the size of the written data in memory in order to decide on closing the writer?</p>
  
  <p>I tried using the byte size of the avro messages that I write to ParquetWriter as an estimate for the Parquet writer file size but that is very different than the parquet writer size because of the different way parquet store data (columnar format). Here's the pseudo code for what I did:</p>
  
  <pre><code>    ParquetWriter parquetWriter = new ParquetWriter(..., BLOCK_SIZE, PAGE_SIZE);    
      long bytesWrittenSofar = 0;
  
      public long getLength(){
          return bytesWrittenSofar;
      }
      public void write(org.apache.avro.generic.GenericRecord record){
          parquetWriter.write(record);
          bytesWrittenSofar += avroToBytes(record).length;
      }
  
      public static byte[] avroToBytes(GenericRecord record){
          GenericDatumWriter&lt;GenericRecord&gt; writer =
              new GenericDatumWriter&lt;GenericRecord&gt;(record.getSchema());
          ByteArrayOutputStream out = new ByteArrayOutputStream();
          BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
          writer.write(record, encoder);
          encoder.flush();
          out.close();
          return out.toByteArray();
      } 
  </code></pre>
  
  <p>It turned out that value I get from getLength() is very different than actual file size of the parquet file. I know that the schema will be added at the end of the file but that is very small. Just to give you an idea, When getLength() reports 130MB, the actual file size is only 80MB.</p>