<p><strong>Context</strong></p>

<p>We have long running kubernetes jobs based on docker containers.
The containers needs resources (eg 15gb memory, 2 cpu) and we use autoscaler to scale up new worker nodes at request.</p>

<p><strong>Scenario</strong></p>

<p>Users can select the version of the docker image to be used for a job, eg 1.0.0, 1.1.0, or even a commit hash of the code the image was build from in test environment.</p>

<p>As we leave the docker tag to be freetext, the user can type a non-existing docker tag. Because of this the job pod comes in ImagePullBackOff state. The pod stays in this state and keeps the resources locked so that they cannot be reused by any other job.</p>

<p><strong>Question</strong></p>

<p>What is the right solution, that can be applied in kubernetes itself, for failing the pod immediately or at least quickly if a pull fails due to a non existing docker image:tag?</p>

<p><strong>Possibilities</strong></p>

<p>I looked into backofflimit. I have set it to 0, but this doesn't fail or remove the job. The resources are of course kept as well.</p>

<p>Maybe they can be killed by a cron job. Not sure how to do so.</p>

<p>Ideally, resources should not even be allocated for a job with an unexisting docker image. But I'm not sure if there is a possibility to easily achieve this.</p>

<p>Any other?</p>

<p>After Looking at your design, I would recommend to add <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="nofollow noreferrer">InitContainer</a> to Job specification to <a href="https://stackoverflow.com/questions/32113330/check-if-imagetag-combination-already-exists-on-docker-hub">check existence</a> of docker images with the given tag. </p>

<p>If the image with the tag doesn't exist in the registry, InitContainer can report an error and fail the Job's Pod by exiting with non-zero exit code. </p>

<p>After that Job's Pod will be <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy" rel="nofollow noreferrer">restarted</a>. After certain amount of attempts Job will get <code>Failed</code> state. By configuring <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#ttl-mechanism-for-finished-jobs" rel="nofollow noreferrer">.spec.ttlSecondsAfterFinished</a> option, failed jobs can be wiped out. </p>

<blockquote>
  <p>If a Pod’s init container fails, Kubernetes repeatedly restarts the Pod until the init container succeeds. However, if the Pod has a restartPolicy of Never, Kubernetes does not restart the Pod.</p>
</blockquote>

<p>If the image exists, InitContainer script exits with zero exit code and the main Job container image is going to be pulled and container starts.</p>
