title: Threadpool multi-queue job dispatch algorithm
body: <p>I'm curious to know if there is a widely accepted solution for managing thread resources in a threadpool given the following scenario/constraints:</p>
  
  <ol>
  <li>Incoming jobs are all of the same
  nature and could be processed by any
  thread in the pool. </li>
  <li>Incoming jobs
  will be 'bucketed' into different
  queues based on some attribute of
  the incoming job such that all jobs
  going to the same bucket/queue MUST
  be processed serially.  </li>
  <li>Some buckets will be less busy than
  others at different points during
  the lifetime of the program.</li>
  </ol>
  
  <p>My question is on the theory behind a threadpool's implementation. What algorithm could be used to efficiently allocate available threads to incoming jobs across all buckets?</p>
  
  <p><strong>Edit</strong>: Another design goal would be to eliminate as much latency as possible between a job being enqueued and it being picked up for processing, assuming there are available idle threads.</p>
  
  <p><strong>Edit2</strong>: In the case I'm thinking of there are a relatively large number of queues (50-100) which have unpredictable levels of activity, but probably only 25% of them will be active at any given time. </p>
  
  <p>The first (and most costly) solution I can think of is to simply have 1 thread assigned to each queue. While this will ensure incoming requests are picked up immediately, it is obviously inefficient. </p>
  
  <p>The second solution is to combine the queues together based on expected levels of activity so that the number of queues is inline with the number of threads in the pool, allowing one thread to be assigned to each queue. The problem here will be that incoming jobs, which otherwise could be processed in parallel, will be forced to wait on each other.</p>
  
  <p>The third solution is to create the maximum number of queues, one for each set of jobs that must be processed serially, but only allocate threads based on the number of queues we expect to be busy at any given time (which could also be adjusted by the pool at runtime). So this is where my question comes in: Given that we have more queues than threads, how does the pool go about allocating idle threads to incoming jobs in the most efficient way possible? </p>
  
  <p>I would like to know if there is a widely accepted approach. Or if there are different approaches - who makes use of which one? What are the advantages/disadvantages, etc? </p>
  
  <p><strong>Edit3</strong>:This might be best expressed in pseudo code.</p>