<p>I need to get a work-item from a work-queue and then sequentially run a series of containers to process each work-item. This can be done using initContainers (<a href="https://stackoverflow.com/a/46880653/94078">https://stackoverflow.com/a/46880653/94078</a>)</p>

<p>What would be the recommended way of restarting the process to get the next work-item?</p>

<ul>
<li>Jobs seem ideal but don't seem to support an infinite/indefinite number of completions.</li>
<li>Using a single Pod doesn't work because initContainers aren't restarted (<a href="https://github.com/kubernetes/kubernetes/issues/52345" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/52345</a>).</li>
<li>I would prefer to avoid the maintenance/learning overhead of a system like argo or brigade.</li>
</ul>

<p>Thanks!</p>

<p>Jobs <em>should</em> be used for working with work queues. When using work queues you should <strong>not set</strong> the <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#jobspec-v1-batch" rel="noreferrer"><code>.spec.comletions</code></a> (or set it to <code>null</code>). In that case Pods will keep getting created until one of the Pods exit successfully. It is a little awkward exiting from the (main) container with a failure state on purpose, but this is the specification. You may set <code>.spec.parallelism</code> to your liking irrespective of this setting; I've set it to <code>1</code> as it appears you do not want any parallelism.</p>

<p>In your question you did not specify what you want to do if the work queue gets empty, so I will give two solutions, one if you want to wait for new items (infinite) and one if want to end the job if the work queue gets empty (finite, but indefinite number of items).</p>

<p>Both examples use redis, but you can apply this pattern to your favorite queue. Note that the part that pops an item from the queue is not safe; if your Pod dies for some reason after having popped an item, that item will remain unprocessed or not fully processed. See the <a href="https://redis.io/commands/rpoplpush#pattern-reliable-queue" rel="noreferrer">reliable-queue pattern</a> for a proper solution.</p>

<p>To implement the sequential steps on each work item I've used <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="noreferrer">init containers</a>. Note that this really is a primitve solution, but you have limited options if you don't want to use some framework to implement a proper pipeline.</p>

<blockquote>
  <p><strong>There is an <a href="https://asciinema.org/a/174054" rel="noreferrer">asciinema</a> if any would like to see this at work without deploying redis, etc.</strong></p>
</blockquote>

<h1>Redis</h1>

<p>To test this you'll need to create, at a minimum, a redis Pod and a Service. I am using the example from <a href="https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/" rel="noreferrer">fine parallel processing work queue</a>. You can deploy that with:</p>

<pre><code>kubectl apply -f https://rawgit.com/kubernetes/website/master/docs/tasks/job/fine-parallel-processing-work-queue/redis-pod.yaml
kubectl apply -f https://rawgit.com/kubernetes/website/master/docs/tasks/job/fine-parallel-processing-work-queue/redis-service.yaml
</code></pre>

<p>The rest of this solution expects that you have a service name <code>redis</code> in the same namespace as your Job and it does not require authentication and a Pod called <code>redis-master</code>.</p>

<h1>Inserting items</h1>

<p>To insert some items in the work queue use this command (you will need bash for this to work):</p>

<pre><code>echo -ne "rpush job "{1..10}"\n" | kubectl exec -it redis-master -- redis-cli
</code></pre>

<h1>Infinite version</h1>

<p>This version waits if the queue is empty thus it will never complete.</p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: primitive-pipeline-infinite
spec:
  parallelism: 1
  completions: null
  template:
    metadata:
      name: primitive-pipeline-infinite
    spec:
      volumes: [{name: shared, emptyDir: {}}]
      initContainers:
      - name: pop-from-queue-unsafe
        image: redis
        command: ["sh","-c","redis-cli -h redis blpop job 0 &gt;/shared/item.txt"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      - name: step-1
        image: busybox
        command: ["sh","-c","echo step-1 working on `cat /shared/item.txt` ...; sleep 5"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      - name: step-2
        image: busybox
        command: ["sh","-c","echo step-2 working on `cat /shared/item.txt` ...; sleep 5"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      - name: step-3
        image: busybox
        command: ["sh","-c","echo step-3 working on `cat /shared/item.txt` ...; sleep 5"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      containers:
      - name: done
        image: busybox
        command: ["sh","-c","echo all done with `cat /shared/item.txt`; sleep 1; exit 1"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      restartPolicy: Never
</code></pre>

<h1>Finite version</h1>

<p>This version stops the job if the queue is empty. Note the trick that the pop init container checks if the queue is empty and all the subsequent init containers <strong>and</strong> the main container immediately exit if it is indeed empty - this is the mechanism that signals Kubernetes that the Job is completed and there is no need to create new Pods for it.</p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: primitive-pipeline-finite
spec:
  parallelism: 1
  completions: null
  template:
    metadata:
      name: primitive-pipeline-finite
    spec:
      volumes: [{name: shared, emptyDir: {}}]
      initContainers:
      - name: pop-from-queue-unsafe
        image: redis
        command: ["sh","-c","redis-cli -h redis lpop job &gt;/shared/item.txt; grep -q . /shared/item.txt || :&gt;/shared/done.txt"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      - name: step-1
        image: busybox
        command: ["sh","-c","[ -f /shared/done.txt ] &amp;&amp; exit 0; echo step-1 working on `cat /shared/item.txt` ...; sleep 5"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      - name: step-2
        image: busybox
        command: ["sh","-c","[ -f /shared/done.txt ] &amp;&amp; exit 0; echo step-2 working on `cat /shared/item.txt` ...; sleep 5"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      - name: step-3
        image: busybox
        command: ["sh","-c","[ -f /shared/done.txt ] &amp;&amp; exit 0; echo step-3 working on `cat /shared/item.txt` ...; sleep 5"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      containers:
      - name: done
        image: busybox
        command: ["sh","-c","[ -f /shared/done.txt ] &amp;&amp; exit 0; echo all done with `cat /shared/item.txt`; sleep 1; exit 1"]
        volumeMounts: [{name: shared, mountPath: /shared}]
      restartPolicy: Never
</code></pre>
