<p>I've been looking for days for a way to set-up a cron-job with a dynamic number of jobs.</p>
<p>I've read all <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-patterns" rel="nofollow noreferrer">these solutions</a> and it seems that, in order to initialise a dynamic number of jobs, I need to do it manually with a script and a job template, but I need it to be automatic.</p>
<p>A bit of context:</p>
<p>I have a database / message queue / whatever can store &quot;items&quot;</p>
<p>I would like to start a job (so a single replica of a container) every 5 minutes to process each item</p>
<p>So, let's say there is a Kafka topic / a db table / a folder containing 5 records / rows / files, I would like Kubernetes to start 5 replicas of the job (with the cron-job) automatically. After 5 minutes, there will be 2 items, so Kubernetes will just start 2 replicas.</p>
<p>The most feasible solution seems to be using a static number of pods and make them process multiple items, but I feel like there is a better way to accomplish my desire keeping it inside Kubernetes that I can't figure due to my lack of experience. 🤔</p>
<p>What would you do to solve this problem?</p>
<p>P.S. Sorry for my English.</p>

Simon
<p>There are two ways I can think of:</p>
<ol>
<li>Using a <code>CronJob</code> that is parallelised (1 work-item/pod or 1+ work-items/pod). This is what you're trying to achieve. Somewhat.</li>
<li>Using a data processing application. This I believe is the recommended approach.</li>
</ol>
<h2>Why and Why Not <code>CronJobs</code></h2>
<p>For (1), there are a few things that I would like to mention. There is no upside to having multiple <code>Job</code>/<code>CronJob</code> items when you are trying to perform the same operation from all of them. You think you are getting parllelism, but not really, you are only increasing management overhead. If your workload grows too large (which it will) there will be too many <code>Job</code> objects in the cluster and the API server will be slowed down drastically.</p>
<p><code>Job</code> and <code>CronJob</code> items are only for stand-alone work items that need to be performed regularly. They are house-keeping tasks. So, selecting <code>CronJobs</code> for data processing is a very bad idea. Even if you run a parallelized set of pods (as provided <a href="https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/" rel="nofollow noreferrer">here</a> and <a href="https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/" rel="nofollow noreferrer">here</a> in the docs like you mentioned), even then, it would be best suited to have a <em><strong>single</strong></em> <code>Job</code> that handles all the pods that are working on the same <code>work-item</code>. So, you should not be thinking of &quot;scaling Jobs&quot; in those terms. Instead, think of scaling <code>Pods</code>. So, if you really want to move ahead with utilizing the <code>Job</code> and <code>CronJob</code> mechanisms, go ahead, the <code>MessageQueue</code> based design is your best bet. And you will have to reinvent a lot of wheels to get it to work (read below why that is the case).</p>
<h2>Recommended Solution</h2>
<p>For (2), I only say this since I see you are trying to perform data processing and doing this with a one-off mechanism like a <code>Job</code> will not be a good idea (<code>Jobs</code> are basically stateless, since they perform an operation that can be repeated simply without  any repercussions). Say you start a pod, it fails processing, how will other pods know that this <code>item</code> was not processed successfully? What if the pod dies, the <code>Job</code> cannot keep track of the <code>items</code> in your data store, since the <code>Job</code> is not aware of the nature of the work you're performing. Therefore, it is natural for you to pursue a solution where the system components are <em><strong>specifically designed</strong></em> for data processing.</p>
<p>You will want to look into a system that can understand the nature of your data, how to keep track of the processing queues that have been finished successfully, how to restart a new <code>Pod</code> with the same <code>item</code> as input, from the <code>Pod</code> that just crashed etc. This is a lot of application/use-case specific functionality that is best served through the means of an <code>operator</code> or a <code>CustomResource</code> and a controller. And obviously, since this is not a new problem, there is a ton of solutions out there that can perform this the best way for you.</p>
<p>The best course of action would be to have that system in place, deployed with the means of a <code>Deployment</code> pattern, where auto-scaling would be enabled and you will achieve real parallelism that will also be best suited for data processing batch jobs.</p>
<p>And remember, when we talk about scaling in Kubernetes, it is <em><strong>always the pods that scale</strong></em>, not containers, not deployments, not services. Always <code>Pods</code>. That is because at the bottom of the chain, there is always a <code>Pod</code> somewhere that is working on something be it a <code>Job</code> that owns it, or a <code>Deployment</code> or a <code>Service</code> a <code>DaemonSet</code> or whatever. And it is obviously a bad idea to have multiple application containers in a Pod due to so many reasons. (<code>side-car</code> and <code>adapter</code> patterns are just helpers, they don't run the application).</p>
<p>Perhaps <a href="https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/" rel="nofollow noreferrer">this blog</a> that discusses data processing in Kubernetes can help.</p>
