Python RQ: pattern for callback <p>I have now a big number of documents to process and am using Python RQ to parallelize the task.</p> <p>I would like a pipeline of work to be done as different operations is performed on each document. For example: <code>A</code> -> <code>B</code> -> <code>C</code> means pass the document to function <code>A</code>, after <code>A</code> is done, proceed to <code>B</code> and last <code>C</code>.</p> <p>However, Python RQ does not seem to support the pipeline stuff very nicely.</p> <p>Here is a simple but somewhat dirty of doing this. In one word, each function along the pipeline call its next function in a nesting way.</p> <p>For example, for a pipeline <code>A</code>-><code>B</code>-><code>C</code>.</p> <p>At the top level, some code is written like this:</p> <p><code>q.enqueue(A, the_doc)</code></p> <p>where q is the <code>Queue</code> instance and in function <code>A</code> there are code like:</p> <p><code>q.enqueue(B, the_doc)</code></p> <p>And in <code>B</code>, there are something like this:</p> <p><code>q.enqueue(C, the_doc)</code></p> <p>Is there any other way more elegant than this? For example some code in <strong>ONE</strong> function:</p> <p><code>q.enqueue(A, the_doc) q.enqueue(B, the_doc, after = A) q.enqueue(C, the_doc, after= B) </code></p> <p><a href="http://python-rq.org/docs/" rel="nofollow">depends_on</a> parameter is the closest one to my requirement, however, running something like:</p> <p><code> A_job = q.enqueue(A, the_doc) q.enqueue(B, depends_on=A_job )</code></p> <p>won't work. As <code>q.enqueue(B, depends_on=A_job )</code> is executed immediately after <code>A_job = q.enqueue(A, the_doc)</code> is executed. By the time B is enqueued, the result from A might not be ready as it takes time to process.</p> <p>PS:</p> <p>If Python RQ is not really good at this, what else tool in Python can I use to achieve the same purpose:</p> <ol> <li>round-robin parallelization</li> <li>pipeline processing support</li> </ol>