title: AWS Redshift Parquet COPY has an incompatible Parquet schema
body: <p>I am writing DataFrame to Redshift using temporary s3 bucket and Parquet as the temporary format. Spark successfully has written data to s3 temp bucket, but Redshift trying to COPY data to warehouse failed with the following error:</p>
  
  <pre><code>error:  S3 Query Exception (Fetch)
   code:      15001
   context:   Task failed due to an internal error. File 'https://s3.amazonaws.com/...../part-00001-de882e65-a5fa-4e52-95fd-7340f40dea82-c000.parquet  has an incompatible Parquet schema for column 's3://bucket-dev-e
   query:     17567
   location:  dory_util.cpp:872
   process:   query0_127_17567 [pid=13630]
  </code></pre>
  
  <p>What am I doing wrong and how to fix it?</p>
  
  <p><strong>UPDATED 1</strong></p>
  
  <p>This is the detailed error:</p>
  
  <pre><code>S3 Query Exception (Fetch). Task failed due to an internal error. 
  File 'https://....d5e6c7a/part-00000-9ca1b72b-c5f5-4d8e-93ce-436cd9c3a7f1-c000.parquet  has an incompatible Parquet schema for column 's3://.....a-45f6-bd9c-d3d70d5e6c7a/manifest.json.patient_dob'. 
  Column type: TIMESTAMP, Parquet schema:\noptional byte_array patient_dob [i:26 d:1 r:0]\n (s3://.......-45f6-bd9c-d3d70d5e6c7a/
  </code></pre>
  
  <p>Apache Spark version <code>2.3.1</code></p>
  
  <p>Also tried to set up the following properties with no luck:</p>
  
  <pre><code>writer
     .option("compression", "none")
     .option("spark.sql.parquet.int96TimestampConversion", "true")
     .option("spark.sql.parquet.int96AsTimestamp", "true")
     .option("spark.sql.parquet.writeLegacyFormat", "true")
  </code></pre>
  
  <p>Where may be the issue?</p>
  
  <p><strong>UPDATED 2</strong></p>
  
  <p>Dataframe <code>patient_dob</code> column type is <code>DateType</code></p>
  
  <p>Redshift <code>patient_dob</code> field type is <code>date</code></p>
  
  <p>S3 select shows the following on <code>patient_dob</code> Parquet field - <code>"patient_dob": "1960-05-28"</code></p>