title: Eventloop design for multithreaded environment with data-queue
body: <p>The description of the problem I am currently trying to find a lispy/working solution for:</p>
  
  <p>A job-queue provides a set of equal (by their code) threads with tasks they shall take care of. If the queue is empty the threads shall wait until a new entry has been made, but I also want to provide a clean shutdown. Therefore even while waiting for the queue, it has to be possible for the mother-thread to set some variable/to call the threads and tell them to shut down. The only reason they would not comply directly should be that the thread is currently evaluating a tasks and therefore busy/not able to make a clean shutdown until the task is done.</p>
  
  <p>I currently got two solutions I am not really convinced by:</p>
  
  <pre><code>(defparameter *kill-yourself* nil)
  
  (defparameter *mutex* (sb-thread:make-mutex))
  
  (defparameter *notify* (sb-thread:make-waitqueue))
  
  #|the queue is thread safe|#
  (defparameter *job-queue* (make-instance 'queue))
  
  
  (defun fill-queue (with-data)
     (fill-stuff-in-queue)
     (sb-thread:with-mutex (*mutex*)
       (sb-thread:condition-notify *notify*)))
  
  
  #|solution A|#
  (with-mutex (*mutex*)
    (do ((curr-job nil))
        (*kill-yourself* nil)
      (if (is-empty *job-queue*)
      (sb-thread:condition-wait *notify* *mutex*)
      (progn
        (setf curr-job (dequeue *job-queue*))
        (do-stuff-with-job)))))
  
  
  #|solution B|#
  (defun helper-kill-yourself-p ()
    (sb-thread:with-mutex (*mutex*)
       *kill-yourself*))
  
  (do ((job (dequeue-* *job-queue* :timeout 0) 
        (dequeue-* *job-queue* :timeout 0)))
          ((if (helper-kill-yourself-p)
           t
                   (sb-thread:with-mutex (*mutex*)
                       (sb-thread:condition-wait *notify* *mutex*)
                       (if (helper-kill-yourself-p)
                            t
                            nil)))
           (progn
             nil))
       (do-stuff-with-job))
  </code></pre>
  
  <p>Both do-loops could be used to start threads. But A would not really work if there is more than one thread (as the mutex would prevent any parallel action taking place), and the B solution looks/is quite dirty as there is the possibility of sidecases where the extracted job is nil. Furthermore I am not really convinced by the halt-condition, as it is too long and seems to complicated.</p>
  
  <p>What would be the proper way to implement a (do)loop which works on data provided by a queue as long as it is supposed to and is also able to sleep as long as there is no new data and as long as it is not supposed to shut down? Last but not least it has to be possible to use this (do)loop in an unlimited amount of multiple parallel threads.</p>