<p>I have a Python process that I want to fire up every <em>n</em> minutes in a Kubernetes cronjob and read a number of messages (say 5) from a queue, and then process/convert some files and run analysis on results based on these queue messages. If the process is still running after <em>n</em> minutes, I don't want to start a new process. In total, I would like a number of these (say 3) of these to be able to run at the same time, however, there can never be more than 3 processes running at the same time. To try and implement this, I tried the following (simplified):</p>
<pre><code>apiVersion: batch/v1
kind: CronJob
metadata:
  name: some-job
  namespace: some-namespace
spec:
  schedule: &quot;*/5 * * * *&quot;
  concurrencyPolicy: &quot;Forbid&quot;
  jobTemplate:
    spec:
      parallelism: 3
      template:
        spec:
          containers:
          - name: job
            image: myimage:tag
            imagePullPolicy: Always
            command: ['python', 'src/run_job.py']
</code></pre>
<p>Now what this amounts to is a maximum of three processes running at the same time due to 'parallelism' being 3, and concurrencyPolicy being &quot;Forbid&quot;, even if the processes go over the 5 minute mark.</p>
<p>The problem I specifically have is that one pod (e.g. pod 1) can take longer than the other two to finish, which means that pod 2 and 3 might finish after a minute, while pod one only finishes after 10 minutes due to processing of larger files from the queue.</p>
<p>Where I thought that <code>parallelism: 3</code> would cause pod 2 and 3 to be deleted and replaced after finishing (when new cron interval hits), they are not and have to wait for pod 1 to finish before starting three new pods when the cron interval hits again.</p>
<p>When I think about it, this functionality makes sense given the specification and meaning of what a cronjob is. However, I would like to know if it would be able to have these pods/processes not be dependent on one another for restart without having to define a duplicate cronjob, all running one process.</p>
<p>Otherwise, maybe I would like to know if it's possible to easily launch more duplicate cronjobs without copying them into multiple manifests.</p>

<p>Duplicate cronjobs seems to be the way to achieve what you are looking for. Produce 3 duplicates with single job at a time. You could template the job manifest and produce multiple as in the following example. The example is not in your problem context, but you can get the idea.
<a href="http://kubernetes.io/docs/tasks/job/parallel-processing-expansion" rel="nofollow noreferrer">http://kubernetes.io/docs/tasks/job/parallel-processing-expansion</a></p>
