title: How to load mixed Parquet schema into DataFrame using Apache Spark?
body: <p>I have a Spark job continuously uploading Parquet files to S3 (with partitions).<br>
  The files all have the same parquet schema.  </p>
  
  <p>One of the field types had been recently changed (from String to long) and so the parquet schema for some of the partitions is mixed.  </p>
  
  <p>Places having mixed data from both types now <strong>fail</strong> to read some of the content.<br>
  While it seems I can execute: <code>sqlContext.read.load(path)</code><br>
  when trying to apply any fetch operation on the DataFrame (<code>collect</code> for example), the operation fails with <code>ParquetDecodingException</code></p>
  
  <p>I intend to migrate the data and re-format it <strong>but fail to read the mixed content into a DataFrame</strong>.<br>
  How can I load the mixed partitions using Apache Spark into DataFrames or any other Spark construct?</p>
  
  <p>Following is the ParquetDecodingException trace:</p>
  
  <pre><code>scala&gt; df.collect
  [Stage 1:==============&gt;        (1 + 3) / 4]
  WARN TaskSetManager: Lost task 1.0 in stage 1.0 (TID 2, 172.1.1.1, executor 0): org.apache.parquet.io.ParquetDecodingException: 
  Can not read value at 1 in block 0 in file 
  s3a://data/parquet/partition_by_day=20180620/partition_by_hour=10/part-00000-6e4f07e4-3d89-4fad-acdf-37054107dc39.snappy.parquet
      at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:243)
      at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227)
      at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
      at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:102)
      at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:166)
      at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:102)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
      at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
      at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:99)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.Long
      at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:105)
  </code></pre>