title: How to handle schema changes in Spark?
body: <p>Consider the following scenario:<br>
  Incremental data gets ingested daily into a HDFS location, and from there I have to read the data using pyspark and find out the latest/active records. 
  Also, I have to handle schema changes in the data, as new fields may get added. </p>
  
  <p>How can I achieve schema comparison and handle schema changes in pyspark?<br>
  How can I handle data which got loaded before the schema changes? </p>
  
  <p>Is the below approach is a good one?</p>
  
  <ol>
  <li>Generate a script to create hive tables on top of HDFS location.</li>
  <li>Then compare the schema of  source table and Hive table using pyspark. If there is schema change use the new schema from source to create the new ddl for table creation. Drop the existing table and create the table with new schema.</li>
  <li>Create a view from the hive tables to get the latest records using primary key and audit column.</li>
  </ol>