<p>New to kubernetes i´m trying to move a current pipeline we have using a queing  system without k8s.</p>

<p>I have a perl script that generates a list of batch jobs (yml files) for each of the samples that i have to process.
Then i run <code>kubectl apply --recursive -f 16S_jobscripts/</code></p>

<p>For example each sample needs to be treated sequentially and go through different processing</p>

<p>Exemple:</p>

<p>SampleA -> clean -> quality -> some_calculation </p>

<p>SampleB -> clean -> quality -> some_calculation </p>

<p>and so on for 300 samples.</p>

<p>So the idea is to prepare all the yml files and run them sequentially. This is working.</p>

<p>BUT, with this approach i need to wait that all samples are processed (let´s say that all the clean jobs need to completed before i run the next jobs quality).</p>

<p>what would be the best approach in such case, run each sample independently ?? how ?</p>

<p>The yml below describe one Sample for one job. You can see that i´m using a counter (mergereads-1 for sample1(A))</p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: merge-reads-1
  namespace: namespace-id-16s
  labels:
    jobgroup: mergereads
spec:
  template:
    metadata:
      name: mergereads-1
      labels:
        jobgroup: mergereads
    spec:
      containers:
        - name: mergereads-$idx
          image: .../bbmap:latest
          command: ['sh', '-c']
          args: ['
          cd workdir &amp;&amp;

          bbmerge.sh -Xmx1200m in1=files/trimmed/1.R1.trimmed.fq.gz in2=files/trimmed/1.R2.trimmed.fq.gz  out=files/mergedpairs/1.merged.fq.gz  merge=t mininsert=300 qtrim2=t minq=27 ratiomode=t &amp;&amp;
          ls files/mergedpairs/ 
          ']

          resources:
            limits:
              cpu: 1
              memory: 2000Mi
            requests:
              cpu: 0.8
              memory: 1500Mi
          volumeMounts:
            - mountPath: '/workdir'
              name: db
      volumes:
        - name: db
          persistentVolumeClaim:
            claimName: workdir
      restartPolicy: Never
</code></pre>

<p>If i understand you correctly you can use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#parallel-jobs" rel="nofollow noreferrer">parallel-jobs</a> with a use of <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-patterns" rel="nofollow noreferrer">Job Patterns</a>.</p>

<blockquote>
  <p>It does support parallel processing of a set of independent but
  related work items.</p>
</blockquote>

<p>Also you can consider using Argo.
<a href="https://github.com/argoproj/argo" rel="nofollow noreferrer">https://github.com/argoproj/argo</a></p>

<blockquote>
  <p>Argo Workflows is an open source container-native workflow engine for
  orchestrating parallel jobs on Kubernetes. Argo Workflows is
  implemented as a Kubernetes CRD (Custom Resource Definition).</p>
</blockquote>

<p>Please let me know if that helps. </p>
