title: Keeping Oracle data in sync with ElasticSearch cluster
body: <p>My solution to sync Oracle data(updated by various systems) into ElasticSearch is to use triggers(insert/update) in the database table and insert event entry[table name and primary keys] into events table.</p>
  
  <p>A listener fetches the events, locks it, fetches the data from Oracle database to index and then prepares JSON document to index into elasticsearch and indexes it using bulk API.</p>
  
  <p>I was asked to redesign it like below,</p>
  
  <ul>
  <li>Listen to events (inserts/updates in Oracle).</li>
  <li>Post the event to SQS.</li>
  <li>SQS listener to fetch records from Oracle database using web service.</li>
  <li>Web service should have APIs to return record for the given primary key and with batching support for queries without primary keys(to fetch records from related tables - to prepare nested documents).</li>
  <li>SQS listener after fetching all necessary records for a given event, prepares the JSON document to index and indexes it in elasticsearch using bulk API.</li>
  </ul>
  
  <p>I don't find the second approach to be the right solution for the near real-time applications, please help me understand the benefits of this approach.</p>