title: Delta/Incremental Load in Hive
body: <p>I have the use case below :</p>
  
  <p>My application has a table having <strong>multiyear data</strong> in <strong>RDBMS</strong> DB. We have used <strong>sqoop</strong> to get data into  HDFS and have loaded into hive table partitioned by <strong>year, month</strong>.</p>
  
  <p>Now, the application updates, and inserts new records into RDBMS Table table daily as well. These updated records can span across history months. Updated records and new insert records can be determined by updated timestamp field (it will have current day timestamp).</p>
  
  <p>Now the problem here is : how to do delta/incremental load hive table daily using these updated records.</p>
  
  <p>-> I know there is a sqoop functionality which allows incremental imports. But, only new incremental import is not enough for us.</p>
  
  <p>Because -</p>
  
  <p>-> I can not directly insert these records (using insert into) in hive table because it will result in duplicate records (updated records).</p>
  
  <p>-> Same way I can not use insert overwrite statement as these are just update and insert records spanning across multiple month. Insert overwrite will delete earlier records.</p>
  
  <p>Of course, easiest option is to get full data using sqoop daily but we don't want to do it as data volume is large.</p>
  
  <p>So , basically we want to fully load only those partitions for which we have received update/insert records.</p>
  
  <p>We are open to explore option at hive or sqoop end. Can you please let us know?</p>
  
  <p>Thanks in advance.</p>