title: AvroRuntimeException: Not a union: {"type":"long","logicalType":"timestamp-millis"}
body: <p>I am trying to save the data from Spark dataframe to HDFS using Avro schema stored in the schema registry. However, I get an error while writing the data:</p>
  
  <pre><code>Caused by: org.apache.avro.AvroRuntimeException: Not a union: {"type":"long","logicalType":"timestamp-millis"}
      at org.apache.avro.Schema.getTypes(Schema.java:299)
      at org.apache.spark.sql.avro.AvroSerializer.org$apache$spark$sql$avro$AvroSerializer$$resolveNullableType(AvroSerializer.scala:229)
      at org.apache.spark.sql.avro.AvroSerializer$$anonfun$3.apply(AvroSerializer.scala:209)
      at org.apache.spark.sql.avro.AvroSerializer$$anonfun$3.apply(AvroSerializer.scala:208)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
      at scala.collection.immutable.List.map(List.scala:296)
      at org.apache.spark.sql.avro.AvroSerializer.newStructConverter(AvroSerializer.scala:208)
      at org.apache.spark.sql.avro.AvroSerializer.&lt;init&gt;(AvroSerializer.scala:51)
      at org.apache.spark.sql.avro.AvroOutputWriter.serializer$lzycompute(AvroOutputWriter.scala:42)
      at org.apache.spark.sql.avro.AvroOutputWriter.serializer(AvroOutputWriter.scala:42)
      at org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:64)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  </code></pre>
  
  <p>What could be the reason for it?</p>
  
  <p>The fields in Avro schema look like this:</p>
  
  <pre><code>{"name":"CreateDate","type":["null",{"type":"long","logicalType":"timestamp-millis"}],"default":null}
  </code></pre>
  
  <p>Here is an example of the date format:</p>
  
  <pre><code>1900-01-01 00:00:00
  </code></pre>
  
  <p>The data type of this field in Spark dataframe:</p>
  
  <pre><code>|-- CreateDate: timestamp (nullable = true)
  </code></pre>
  
  <p>This is the way I write the data:</p>
  
  <pre><code>dataDF.write
    .mode("append")
    .format("avro")
    .option(
      "avroSchema",
      SchemaRegistry.getSchema(
        schemaRegistryConfig.url,
        schemaRegistryConfig.dataSchemaSubject,
        schemaRegistryConfig.dataSchemaVersion))
    .save(hdfsURL)
  </code></pre>