title: Threads processing a batch job in servlet enviornment
body: <p>I have a Spring-MVC, Hibernate, (Postgres 9 db) Web app. An admin user can send in a request to process nearly 200,000 records (each record collected from various tables via joins). Such operation is requested on a weekly or monthly basis (OR whenever the data reaches to a limit of around 200,000/100,000 records). On the database end, i am correctly implementing batching.</p>
  <ul>
  <li><p>PROBLEM: Such a long running request holds up the server thread and that causes the the normal users to suffer.</p>
  </li>
  <li><p>REQUIREMENT: The high response time of this request is not an issue. Whats required is not make other users suffer because of this time consuming process.</p>
  </li>
  <li><p>MY SOLUTION:</p>
  <p>Implementing threadpool using Spring taskExecutor abstraction. So i can initialize my threadpool with say 5 or 6 threads and break the 200,000 records into smaller chunks say of size 1000 each. I can queue in these chunks. To further allow the normal users to have a faster db access, maybe I can make every runnable thread sleep for 2 or 3 secs.
  Advantages of this approach i see is: Instead of executing a huge db interacting request in one go, we have a asynchronous design spanning over a larger time. Thus behaving like multiple normal user requests.</p>
  </li>
  </ul>
  <p>Can some experienced people please give their opinion on this?
  I have also read about implementing the same beahviour with a Message Oriented Middleware like JMS/AMQP OR Quartz Scheduling. But frankly speaking, i think internally they are also gonna do the same thing i.e making a thread pool and queueing in the jobs. So why not go with the Spring taskexecutors instead of adding a completely new infrastructure in my web app just for this feature?</p>
  <p>Please share your views on this and let me know if there is other better ways to do this?
  Once again: the time to completely process all the records in not a concern, whats required is that normal users accessing the web app during that time should not suffer in any way.</p>