title: Matching huge tables data in Distributed Environment
body: <p>I have a shell script job which provides the input data that needs to be match and suppressed with n number of huge tables (Table size varies from 20 million to 150 million records). Multiple job may be submitted at the same time</p>
  
  <p>Input data size varies from 500k to 3MM records.
  No of tables varies from 5 to 40.
  Huge tables data will be updated on real-time basis;
  Matching Column will have index created.</p>
  
  <p>Currently I am doing this task using Mysql on Linux Sever (Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz and Ram 128Gb) which is taking long time. I have provided the shell script which accepts input data file as argument and hit the database to execute below queries and get the clean data back.I am looking for the better approach to this problem.</p>
  
  <pre class="lang-sql prettyprint-override"><code>delete a from inputTableName a, table1 b where a.email=b.email;
  delete a from inputTableName a, table2 b where a.email=b.email;
  delete a from inputTableName a, table3 b where a.email=b.email;
  delete a from inputTableName a, table4 b where a.email=b.email;
  delete a from inputTableName a, table5 b where a.email=b.email;
  delete a from inputTableName a, table6 b where a.email=b.email;
  </code></pre>
  
  <p>Need the result in 3 to 5 minutes per job. Wanted to know if this problem can be solved using distributed approach. I am open to use any language or technology. </p>