title: Apache Avro schema validation in Apache Flume
body: <p>After reading about Apache Flume and the benefits it provides in terms of handling client events I decided it was time to start looking into this in more detail. Another great benefit appears to be that it can handle Apache Avro objects :-) However, I am struggle to understand how the Avro schema is used to validate Flume events received.</p>
  <p>To help understand my problem in more detail I have provided code snippets below;</p>
  <h1>Avro schema</h1>
  <p>For the purpose of this post I am using a sample schema defining a nested <code>Object1</code> record with 2 fields.</p>
  <pre><code>{
    &quot;namespace&quot;: &quot;com.example.avro&quot;,
    &quot;name&quot;: &quot;Example&quot;,
    &quot;type&quot;: &quot;record&quot;,
    &quot;fields&quot;: [
      {
        &quot;name&quot;: &quot;object1&quot;,
        &quot;type&quot;: {
          &quot;name&quot;: &quot;Object1&quot;,
          &quot;type&quot;: &quot;record&quot;,
          &quot;fields&quot;: [
            {
              &quot;name&quot;: &quot;value1&quot;,
              &quot;type&quot;: &quot;string&quot;
            },
            {
              &quot;name&quot;: &quot;value2&quot;,
              &quot;type&quot;: &quot;string&quot;
            }
          ]
        }
      }
    ]
  }
  </code></pre>
  <h1>Embedded Flume agent</h1>
  <p>Within my Java project I am currently using the Apache Flume embedded agent as detailed below;</p>
  <pre><code>public static void main(String[] args) {
      final Event event = EventBuilder.withBody(&quot;Test&quot;, Charset.forName(&quot;UTF-8&quot;));
  
      final Map&lt;String, String&gt; properties = new HashMap&lt;&gt;();
      properties.put(&quot;channel.type&quot;, &quot;memory&quot;);
      properties.put(&quot;channel.capacity&quot;, &quot;100&quot;);
      properties.put(&quot;sinks&quot;, &quot;sink1&quot;);
      properties.put(&quot;sink1.type&quot;, &quot;avro&quot;);
      properties.put(&quot;sink1.hostname&quot;, &quot;192.168.99.101&quot;);
      properties.put(&quot;sink1.port&quot;, &quot;11111&quot;);
      properties.put(&quot;sink1.batch-size&quot;, &quot;1&quot;);
      properties.put(&quot;processor.type&quot;, &quot;failover&quot;);
  
      final EmbeddedAgent embeddedAgent = new EmbeddedAgent(&quot;TestAgent&quot;);
      embeddedAgent.configure(properties);
      embeddedAgent.start();
  
      try {
          embeddedAgent.put(event);
      } catch (EventDeliveryException e) {
          e.printStackTrace();
      }
  }
  </code></pre>
  <p>In the above example I am creating a new Flume event with &quot;Test&quot; defined as the event body sending events to a separate Apache Flume agent running inside a VM (192.168.99.101).</p>
  <h1>Remote Flume agent</h1>
  <p>As described above I have configured this agent to receive events from the embedded Flume agent. The Flume configuration for this agent looks like;</p>
  <pre><code># Name the components on this agent
  hello.sources = avroSource
  hello.channels = memoryChannel
  hello.sinks = loggerSink
  
  # Describe/configure the source
  hello.sources.avroSource.type = avro
  hello.sources.avroSource.bind = 0.0.0.0
  hello.sources.avroSource.port = 11111
  hello.sources.avroSource.channels = memoryChannel
  
  # Describe the sink
  hello.sinks.loggerSink.type = logger
  
  # Use a channel which buffers events in memory
  hello.channels.memoryChannel.type = memory
  hello.channels.memoryChannel.capacity = 1000
  hello.channels.memoryChannel.transactionCapacity = 1000
  
  # Bind the source and sink to the channel
  hello.sources.avroSource.channels = memoryChannel
  hello.sinks.loggerSink.channel = memoryChannel
  </code></pre>
  <p>And I am executing the following command to launch the agent;</p>
  <pre><code>./bin/flume-ng agent --conf conf --conf-file ../sample-flume.conf --name hello -Dflume.root.logger=TRACE,console -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true
  </code></pre>
  <p>When I execute the Java project main method I see the &quot;Test&quot; event is passed through to my logger sink with the following output;</p>
  <pre><code>2019-02-18 14:15:09,998 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 54 65 73 74                                     Test }
  </code></pre>
  <p>However, it is unclear to me exactly where I should configure the Avro schema to ensure that only valid events are received and processed by Flume. Can someone please help me understand where I am going wrong? Or, if I have misunderstood the intention of how Flume is designed to convert Flume events into Avro events?</p>
  <p>In addition to the above I have also tried using the Avro RPC client after changing the Avro schema to specify a protocol talking directly to my remote Flume agent, but when I attempt to send events I see the following error;</p>
  <pre><code>Exception in thread &quot;main&quot; org.apache.avro.AvroRuntimeException: Not a remote message: test
      at org.apache.avro.ipc.Requestor$Response.getResponse(Requestor.java:532)
      at org.apache.avro.ipc.Requestor$TransceiverCallback.handleResult(Requestor.java:359)
      at org.apache.avro.ipc.Requestor$TransceiverCallback.handleResult(Requestor.java:322)
      at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.messageReceived(NettyTransceiver.java:613)
      at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:595)
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
      at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
      at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
      at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
      at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
      at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
      at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
      at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
      at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  <p>My goal is that I am able to ensure that events populated by my application conforms to the Avro schema generated to avoid invalid events being published. I would prefer I achieve this using the embedded Flume agent, but if this is not possible then I would consider using the Avro RPC approach talking directly to my remote Flume agent.</p>
  <p>Any help / guidance would be a great help. Thanks in advance.</p>
  <h1>UPDATE</h1>
  <p>After further reading I wonder if I have misunderstood the purpose of Apache Flume. I originally thought this could be used to automatically create Avro events based on the data / schema, but now wondering if the application should assume responsibility for producing Avro events which will be stored in Flume according to the channel configuration and sent as a batch via the sink (in my case a Spark Streaming cluster).</p>
  <p>If the above is correct then I would like to know whether Flume is required to know about the schema or just my Spark Streaming cluster which will eventually process this data? If Flume is required to know about the schema then can you please provide details of how this can be achieved?</p>
  <p>Thanks in advance.</p>