title: AVRO schema evolution adding optional column with default fails deserialization
body: <p>When reading through avro documentation, for example [1], I understood, that schema evolution is supported, and if I added column with specified default, it should be backwards compatible (and even forward when I remove it again). Sounds great, so I added column defined as:</p>
  
  <pre><code>        {
            "name": "newColumn",
            "type": ["null","string"],
            "default": null,
            "doc": "something wrong"
          }
  </code></pre>
  
  <p>and try to consumer some topic having this schema from beginning, it fails with message:</p>
  
  <pre><code>Caused by: java.lang.ArrayIndexOutOfBoundsException: 5
      at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:424)
      at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
      at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
      at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
      at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
      at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
      at tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83)
  </code></pre>
  
  <p>to give a little bit more information. Avro schema defines one top level type, having 2 fields. String describing type of message, and union of N types. All N-1, non-modified types can be read, but one updated with optional, default-having column cannot be read. I'm not sure if this design is strictly speaking correct, but that's not the point (feel free to criticise and recommend better approach!). I'm after schema evolution, which <em>seems</em> not to be working. </p>
  
  <p>Am I doing something wrong?</p>
  
  <p>[1] <a href="https://docs.oracle.com/database/nosql-12.1.3.4/GettingStartedGuide/schemaevolution.html#changeschema-rules" rel="nofollow noreferrer">https://docs.oracle.com/database/nosql-12.1.3.4/GettingStartedGuide/schemaevolution.html#changeschema-rules</a></p>
  
  <p>EDIT:
  and if we alter type definition to: </p>
  
  <pre><code>"type": "string",
  "default": ""
  </code></pre>
  
  <p>it still does not work and generated error is:</p>
  
  <pre><code>Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -1
      at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336)
      at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263)
      at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
      at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
      at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
      at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
      at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
      at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
      at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
      at tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83)
  </code></pre>
  
  <p>what code does lead to given failures:</p>
  
  <pre><code>BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(avro, (BinaryDecoder)null);
  GenericRecord record = (GenericRecord)(new GenericDatumReader(schema)).read((Object)null, binaryDecoder);
  </code></pre>