<p>I'm running AWS EKS, running on Fargate, and using Kubernetes to orchestrate multiple cron jobs. I spin roughly 1000 pods up and down over the course of a day.</p>
<p>Very seldomly(once every 3 weeks) one of the pods gets stuck in ContainerCreating and just hangs there and because I have concurrency disabled that particular job will never run. The fix is simply terminating the job or the pod and having it restart but this is a manual intervention.</p>
<p>Is there a way to get a pod to terminate or restart, if it takes too long to create?</p>
<p>The reason for the pod getting stuck varies quite a bit. A solution would need to be general. It can be a time based solution as all the pods are running the same code with different configurations so the startup time is relatively consistent.</p>

Damian Jacobs
<p>Sadly there is no mecanism to stop a job if it fail at image pulling or container creating. I also <a href="https://stackoverflow.com/questions/72041200/kubernetes-limit-number-of-retry">tried to do</a> what you are trying to achieve.</p>
<p>You can set a <code>backoffLimit</code> inside your template. But it won't handle the number of retries during <code>containerCreating</code>, only while running.</p>
<p>What you can do is a script that makes <code>describes</code> of each pods in namespace. And try to parse it and restart the pod if it is stuck in <code>containerCreating</code>.</p>
<p>Or try to debug/trace what is causing this. <code>kubectl describe pods</code> to get info when your pod is in <code>containerCreating</code>.</p>
