title: Java ExecutorService - sometimes slower than sequential processing?
body: <p>I'm writing a simple utility which accepts a collection of Callable tasks, and runs them in parallel. The hope is that the total time taken is little over the time taken by the longest task. The utility also adds some error handling logic - if any task fails, and the failure is something that can be treated as "retry-able" (e.g. a timeout, or a user-specified exception), then we run the task directly.</p>
  
  <p>I've implemented this utility around an ExecutorService. There are two parts:</p>
  
  <ol>
  <li>submit() all the Callable tasks to the ExecutorService, storing the Future objects.</li>
  <li>in a for-loop, get() the result of each Future. In case of exceptions, do the "retry-able" logic.</li>
  </ol>
  
  <p>I wrote some unit tests to ensure that using this utility is faster than running the tasks in sequence. For each test, I'd generate a certain number of Callable's, each essentially performing a Thread.sleep() for a random amount of time within a bound. I experimented with different timeouts, different number of tasks, etc. and the utility seemed to outperform sequential execution.</p>
  
  <p>But when I added it to the actual system which needs this kind of utility, I saw results that were very variable - sometimes the parallel execution was faster, sometimes it was slower, and sometimes it was faster, but still took a lot more time than the longest individual task.</p>
  
  <p>Am I just doing it all wrong? I know ExecutorService has invokeAll() but that swallows the underlying exceptions. I also tried using a CompletionService to fetch task results in the order in which they completed, but it exhibited more or less the same behavior. I'm reading up now on latches and barriers - is this the right direction for solving this problem?</p>