title: Parquet-MR AvroParquetWriter - how to convert data to Parquet (with Specific Mapping)
body: <p>I'm working on a tool for converting data from a homegrown format to Parquet and JSON (for use in different settings with Spark, Drill and MongoDB), using Avro with Specific Mapping as the stepping stone. I have to support conversion of new data on a regular basis and on client machines which is why I try to write my own standalone conversion tool with a (Avro|Parquet|JSON) switch instead of using Drill or Spark or other tools as converters as I probably would if this was a one time job. I'm basing the whole thing on Avro because this seems like the easiest way to get conversion to Parquet and JSON under one hood.</p>
  
  <p>I used Specific Mapping to profit from static type checking, wrote an IDL, converted that to a schema.avsc, generated classes and set up a sample conversion with specific constructor, but now I'm stuck configuring the writers. All Avro-Parquet conversion examples I could find [0] use AvroParquetWriter with deprecated signatures (mostly: <code>Path file, Schema schema</code>) and Generic Mapping. </p>
  
  <p>AvroParquetWriter has only one none-deprecated Constructor, with this signature:  </p>
  
  <pre><code>AvroParquetWriter(
      Path file, 
      WriteSupport&lt;T&gt; writeSupport,
      CompressionCodecName compressionCodecName,
      int blockSize, 
      int pageSize, 
      boolean enableDictionary,
      boolean enableValidation, 
      WriterVersion writerVersion,
      Configuration conf
  )
  </code></pre>
  
  <p>Most of the parameters are not hard to figure out but <code>WriteSupport&lt;T&gt; writeSupport</code> throws me off. I can't find any further documentation or an example.<br>
  Staring at the source of AvroParquetWriter I see <code>GenericData model</code> pop up a few times but only one line mentioning <code>SpecificData</code>: <code>GenericData model = SpecificData.get();</code>.</p>
  
  <p>So I have a few questions:</p>
  
  <p>1) Does AvroParquetWriter not support Avro Specific Mapping? Or does it by means of that <code>SpecificData.get()</code> method? The comment "Utilities for generated Java classes and interfaces." over 'SpecificData.class` seems to suggest that but how exactly should I proceed?</p>
  
  <p>2) What's going on in the AvroParquetWriter constructor, is there an example or some documentation to be found somewhere? </p>
  
  <p>3) More specifically: the signature of the WriteSupport method asks for 'Schema avroSchema' and 'GenericData model'. What does <code>GenericData model</code> refer to? Maybe I'm not seeing the forest because of all the trees here...</p>
  
  <p>To give an example of what I'm aiming for, my central piece of Avro conversion code currently looks like this:</p>
  
  <pre><code>DatumWriter&lt;MyData&gt; avroDatumWriter = new SpecificDatumWriter&lt;&gt;(MyData.class);
  DataFileWriter&lt;MyData&gt; dataFileWriter = new DataFileWriter&lt;&gt;(avroDatumWriter);
  dataFileWriter.create(schema, avroOutput);
  </code></pre>
  
  <p>The Parquet equivalent currently looks like this:</p>
  
  <pre><code>AvroParquetWriter&lt;SpecificRecord&gt; parquetWriter = new AvroParquetWriter&lt;&gt;(parquetOutput, schema);
  </code></pre>
  
  <p>but this is not more than a beginning and is modeled after the examples I found, using the deprecated constructor, so will have to change anyway.</p>
  
  <p>Thanks,<br>
  Thomas</p>
  
  <p>[0] Hadoop - The definitive Guide, O'Reilly, <a href="https://gist.github.com/hammer/76996fb8426a0ada233e" rel="nofollow">https://gist.github.com/hammer/76996fb8426a0ada233e</a>, <a href="http://www.programcreek.com/java-api-example/index.php?api=parquet.avro.AvroParquetWriter" rel="nofollow">http://www.programcreek.com/java-api-example/index.php?api=parquet.avro.AvroParquetWriter</a>   </p>