title: Handling empty arrays in pySpark (optional binary element (UTF8) is not a group)
body: <p>I have a json-like structure in spark which looks as follows:</p>
  
  <pre><code>&gt;&gt;&gt; df = spark.read.parquet(good_partition_path)
  id: string
  some-array: array
      element: struct
          array-field-1: string
          array-field-2: string
  </code></pre>
  
  <p>depending on the partition, <code>some-array</code> might be an empty array for all <code>id</code>'s. When this happend spark infers the following schema:</p>
  
  <pre><code>&gt;&gt;&gt; df = spark.read.parquet(bad_partition_path)
  id: string
  some-array: array
      element: string
  </code></pre>
  
  <p>Of course that's a problem if I want to read multiple partitions because spark cannot merge schemas. I have tried to define manually the schema so there should be no problem</p>
  
  <pre><code>&gt;&gt;&gt; df = spark.read.schema(good_schema).parquet(bad_partition_path)
  id: string
  some-array: array
      element: struct
          array-field-1: string
          array-field-2: string
  </code></pre>
  
  <p>So far so good, but when I try to actually collect the data I get an error:</p>
  
  <pre><code>&gt;&gt;&gt; df.head(5)
  # Long error message
  Caused by: java.lang.ClassCastException: optional binary element (UTF8) is not a group
  </code></pre>
  
  <p>I don't understand why this fails. There should be no incompatibility cause by the schema. In case you wonder, collecting the data without specifying the schema works.</p>
  
  <pre><code>&gt;&gt;&gt; df = spark.read.parquet(bad_partition_path)
  id: string
  some-array: array
      element: string   # infers wrong schema
  &gt;&gt;&gt; df.head(5)
  [Row(...)] # actually works
  </code></pre>
  
  <h1>Edit</h1>
  
  <p>Here there's a reproducible example in python </p>
  
  <pre><code>from pyspark.sql.types import *
  
  
  myschema = StructType([
     StructField('id', StringType())
   , StructField( 'some-array'
                , ArrayType(StructType([
                    StructField('array-field-1', StringType())
                  , StructField('array-field-2', StringType())
                  ])
                ))
    ])
  
  path_writeKO = "path/to/parquet"
  jsonKO = '{"id": "OK", "some-array": []}'
  dfKO = sc.parallelize([jsonKO])
  dfKO = spark.read.json(dfKO)
  dfKO.write.parquet(path_writeKO) # write without schema
  
  read_error = spark.read.schema(myschema).parquet(path_writeKO) # read with schema
  read_error.collect() # Fails!!
  
  </code></pre>