title: How can I use the AvroParquetWriter and write to S3 via the AmazonS3 api?
body: <p>I am currently using the code below to write parquet via Avro.  This code writes it to a file system but I want to write to S3. </p>
  
  <pre><code>try {
      StopWatch sw = StopWatch.createStarted();
      Schema avroSchema = AvroSchemaBuilder.build("pojo", message.getTransformedMessage().get(0));
      final String parquetFile = "parquet/data.parquet";
      final Path path = new Path(parquetFile);
  
      ParquetWriter writer = AvroParquetWriter.&lt;GenericData.Record&gt;builder(path)
          .withSchema(avroSchema)
          .withConf(new org.apache.hadoop.conf.Configuration())
          .withCompressionCodec(CompressionCodecName.SNAPPY)
          .withWriteMode(Mode.OVERWRITE)//probably not good for prod. (overwrites files).
          .build();
  
      for (Map&lt;String, Object&gt; row : message.getTransformedMessage()) {
        StopWatch stopWatch = StopWatch.createStarted();
        final GenericRecord record = new GenericData.Record(avroSchema);
        row.forEach((k, v) -&gt; {
          record.put(k, v);
        });
        writer.write(record);
      }
      //todo:  Write to S3.  We should probably write via the AWS objects.  This does not show that.
      //https://stackoverflow.com/questions/47355038/how-to-generate-parquet-file-using-pure-java-including-date-decimal-types-an
      writer.close();
      System.out.println("Total Time: " + sw);
  
    } catch (Exception e) {
      //do somethign here.  retryable?  non-retryable?  Wrap this excetion in one of these?
      transformedParquetMessage.getOriginalMessage().getMetaData().addException(e);
    }
  </code></pre>
  
  <p>This writes to a file fine, but how do I get it to stream it into the AmazonS3 api?  I have found some code on the web using the Hadoop-aws jar, but that requires some Windows exe files to work and, of course, we want to avoid that.  Currently I am using only:</p>
  
  <pre><code> &lt;dependency&gt;
    &lt;groupId&gt;org.apache.avro&lt;/groupId&gt;
    &lt;artifactId&gt;avro&lt;/artifactId&gt;
    &lt;version&gt;1.9.2&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
    &lt;groupId&gt;org.apache.parquet&lt;/groupId&gt;
    &lt;artifactId&gt;parquet-avro&lt;/artifactId&gt;
    &lt;version&gt;1.8.1&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
    &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
    &lt;artifactId&gt;hadoop-core&lt;/artifactId&gt;
    &lt;version&gt;1.2.1&lt;/version&gt;
  &lt;/dependency&gt;
  </code></pre>
  
  <p><strong>So the question is, is there a way to intercept the output stream on the AvroParquetWriter so I can stream it to S3?</strong>  The main reason I want to do this is for retries.  S3 automagically retries up to 3 times.  This would help us out a lot.</p>