title: Avro schema doesn't honor backward compatibilty
body: <p>I have this avro schema</p>
  
  <pre><code>{
   "namespace": "xx.xxxx.xxxxx.xxxxx",
   "type": "record",
   "name": "MyPayLoad",
   "fields": [
       {"name": "filed1",  "type": "string"},
       {"name": "filed2",     "type": "long"},
       {"name": "filed3",  "type": "boolean"},
       {
            "name" : "metrics",
            "type": 
            {
               "type" : "array", 
               "items": 
               { 
                   "name": "MyRecord", 
                   "type": "record", 
                   "fields" : 
                       [                         
                         {"name": "min", "type": "long"}, 
                         {"name": "max", "type": "long"}, 
                         {"name": "sum", "type": "long"}, 
                         {"name": "count", "type": "long"}
                       ]
               } 
            }
       }
    ]
  }
  </code></pre>
  
  <p>Here is the code which we use to parse the data</p>
  
  <pre><code>public static final MyPayLoad parseBinaryPayload(byte[] payload) {
          DatumReader&lt;MyPayLoad&gt; payloadReader = new SpecificDatumReader&lt;&gt;(MyPayLoad.class);
          Decoder decoder = DecoderFactory.get().binaryDecoder(payload, null);
          MyPayLoad myPayLoad = null;
          try {
              myPayLoad = payloadReader.read(null, decoder);
          } catch (IOException e) {
              logger.log(Level.SEVERE, e.getMessage(), e);
          }
  
          return myPayLoad;
      }
  </code></pre>
  
  <p>Now i want to add one more field int the schema so the schema looks like below</p>
  
  <pre><code> {
   "namespace": "xx.xxxx.xxxxx.xxxxx",
   "type": "record",
   "name": "MyPayLoad",
   "fields": [
       {"name": "filed1",  "type": "string"},
       {"name": "filed2",     "type": "long"},
       {"name": "filed3",  "type": "boolean"},
       {
            "name" : "metrics",
            "type": 
            {
               "type" : "array", 
               "items": 
               { 
                   "name": "MyRecord", 
                   "type": "record", 
                   "fields" : 
                       [                         
                         {"name": "min", "type": "long"}, 
                         {"name": "max", "type": "long"}, 
                         {"name": "sum", "type": "long"}, 
                         {"name": "count", "type": "long"}
                       ]
               } 
            }
       }
       {"name": "agentType",  "type": ["null", "string"], "default": "APP_AGENT"}
    ]
  }
  </code></pre>
  
  <p>Note the filed added and also the default is defined. The problem is that if we receive the data which was written using the older schema i get this error</p>
  
  <pre><code>java.io.EOFException: null
      at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148) ~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139) ~[avro-1.7.4.jar:1.7.4]
      at com.appdynamics.blitz.shared.util.XXXXXXXXXXXXX.parseBinaryPayload(BlitzAvroSharedUtil.java:38) ~[blitz-shared.jar:na]
  </code></pre>
  
  <p>What i understood from <a href="https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html" rel="noreferrer">this</a> document that this should have been backward compatible but somehow that doesn't seem to be the case. Any idea what i am doing wrong?</p>