title: Custom ETL Architecture Performance Issues
body: <p>I am working on internal tool that loads data from multiple sources and apply transformations and publish Data to network.</p>
  
  <p>I am having performance Issues in transforming and publishing data to sql server.</p>
  
  <p>right now we store intermediate data in lucene. which is having issues on updating.</p>
  
  <p>what are the other alternatives to store intermediate data, which has to be fast and easy to update.</p>
  
  <p>we are doing all this in single thread.I knew we have to multiple threads for better performance but not handling threads in a proper way leads to more problems. so we are using single thread.</p>