<p>I have a server I wrote that allows us to "Kick off" processing and or manage CRON jobs to do the processing on different schedules.</p>

<p>If I'm running my container in a horizontally scalable way how to I recover from crashed containers or inform new containers that a CRON job is being handled by another container?</p>

<p>Should I store my cron job definitions and their state in a database? If so how do I record state if a container crashes and does not have the opportunity to write state to the DB?</p>

<p>Should I break out my container definition into a separate container and only ever run one?</p>

<p>If you want your application code to be stateless, then yes, you will need to store it in a database to be resilient to containers crashing.</p>

<p>However, I think you're asking how other containers will pick up and retry the failed cron jobs if a container crashes. You're now looking at designing a distributed job scheduler. Rolling your own is a significant amount of work, and there are already <a href="https://github.com/search?q=distributed+cron" rel="nofollow noreferrer">many off-the-shelf</a> solutions out there.</p>

<p>Thankfully, you're already running a distributed job scheduler, Kubernetes! You could take advantage of Kubernetes <a href="https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/" rel="nofollow noreferrer">CronJob</a> feature. If you <a href="https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod" rel="nofollow noreferrer">configure your application to be able to talk to the Kubernetes API</a>, your application can create CronJob objects, and leave the rest to the scheduler.</p>
