title: Java task queue, threadpool and threads with callback to inform when a new task can start
body: <p>I have tried to find a way how to load several web pages in multiple threads with a certain maximum limit of threads, in a way such that a new page is loaded when one finish. There should also be another post-processing threads for the loaded content after the page is downloaded so that the whole process is chained.</p>
  <p>How I'd like to make it:</p>
  <ul>
  <li><p>Task queue holds the pages that should be downloaded</p>
  </li>
  <li><p>Threadpool has a certain number of threads downloading the pages in the task queue (loading the pages take some time so the number of
  threads can be much higher than the number of the cpu cores)</p>
  </li>
  <li><p>When a page's download is completed, the thread should notify this so that a new task from the queue can be started instead</p>
  </li>
  <li><p>When a page's download is completed, the content should be transferred to another task queue for the post-processing</p>
  </li>
  <li><p>Another threadpool has as many threads as the cpu has cores (guess it is fastest to have only one thread per core for the post-processing), this threadpool perform post-processing on the downloaded pages.</p>
  </li>
  <li><p>When a page's post-processing is completed, the thread should notify it so that another page in the queue can be post-processed</p>
  </li>
  <li><p>When all pages has been downloaded (the queue is empty), the first threadpool can be shutdown, the other threadpool can be shutdown when both task queues are empty (all pages has been downloaded and post-processed)</p>
  </li>
  </ul>
  <p>I have something like:</p>
  <pre><code>            for (int j = 0; j &lt; threads.length; j++) {
              threads[j].start();
          }
  
          for (int j = 0; j &lt; threads.length; j++) {
              threads[j].join();
          }
  </code></pre>
  <p>But this way all pages to load are in separate threads at the same time and I want to limit the number of threads. More importantly I want to reuse the threads and make a thread do the next task when one task is finished. I could do that with a while loop, but this is what I'm trying to avoid, I don't want a while loop to check if the queue has more tasks and if a thread is free. Is it possible to use some kind of callback, so that the thread tells back to the pool that is is completed and returns the data.
  I also want the downloading tasks to ~store the content in a data structure and add it futher to the post-processing task queue.</p>
  <p>The best resources I found so far is:
  <a href="http://tutorials.jenkov.com/java-concurrency/thread-pools.html" rel="nofollow noreferrer">Thread pools</a>
  <a href="https://coderanch.com/t/389877/java/java-callback" rel="nofollow noreferrer">Callback</a></p>
  <p>But I don't know if it even is possible to create it the way I want. I' stuck in thinking about function pointers.</p>