title: Why are new columns added to parquet tables not available from glue pyspark ETL jobs?
body: <p>We've been exploring using Glue to transform some JSON data to parquet. One scenario we tried was adding a column to the parquet table. So partition 1 has columns [A] and partition 2 has columns [A,B]. Then we wanted to write further Glue ETL jobs to aggregate the parquet table but the new column was not available. Using <code>glue_context.create_dynamic_frame.from_catalog</code> to load the dynamic frame our new column was never in the schema.</p>
  
  <p>We tried several configurations for our table crawler. Using a single schema for all partitions, single schema for s3 path, schema per partition. We could always see the new column in the Glue table data but it was always null if we queried it from a Glue job using pyspark. The column was in the parquet when we downloaded some samples and available for querying via Athena.</p>
  
  <p>Why are the new columns not available to pyspark?</p>