<p>I have a Kubernetes cluster running using EKS (Elastic Kubernetes Service) and ECR (Elastic Container Repository) on AWS. One specific deployment of mine runs fine for the first two/three restarts before then always initialising a CrashLoopBackOff on image pull, waiting for the length of the Back Off and then running fine, before repeating the process.</p>
<p>These pods consist of a docker container which waits for a message from a message queue, runs a process, then the docker container stops at which point the deployment will restart the container, always pulling the container from ECR.</p>
<p>As these pods are intended to handle a lot of traffic and have a short runtime (~1-30 seconds), having each pod immediately enter CrashLoopBackOff on pull and then wait for five minutes before actually running is annoying with a lot of waiting time.</p>
<p>I've had a look around for any answers to this, but all the questions I've seen describe cases where CrashLoopBackOff continues to run indefinitely, rather than a pod entering CrashLoopBackOff then running successfully once the wait time has finished.</p>
<p>I've checked the logs for the pods which have this issue and there is nothing there which indicates any errors. I'm wondering if there is a way to &quot;pause&quot; the container after it is pulled, to ensure it is up and running correctly before the docker command is actually run? Or any other way to delay CrashLoopBackOff for a configurable amount of seconds? I've added &quot;sleep 15;&quot; to the start of my docker container command, but that hasn't helped the issue.</p>
<p>Deployment Yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: piml-xgboost
spec:
  replicas: 5
  selector:
    matchLabels:
      app: piml-xgboost
  template:
    metadata:
      labels: 
        app: piml-xgboost
    spec:
      serviceAccountName: cluster-service-account
      containers:
      - name: piml-unet
        image: 'ecr_path'
        imagePullPolicy: &quot;Always&quot;
        resources:
          requests:
            memory: &quot;500Mi&quot;
          limits:
            memory: &quot;4Gi&quot;
        env:
        - name: BROKER_URL
          value: 'amqp_broker_url'
        - name: QUEUE
          value: 'amqp_queue'
        - name: method
          value: xgboost
        - name: k8s
          value: 'True'
</code></pre>
<p>Typical 'kubectl get pods' output:</p>
<pre><code>NAME                                    READY   STATUS             RESTARTS          AGE
piml-xgboost-77d48f9db8-5txmz           0/1     CrashLoopBackOff   959 (2m51s ago)   3d21h
piml-xgboost-77d48f9db8-gs542           0/1     CrashLoopBackOff   532 (108s ago)    2d1h
piml-xgboost-77d48f9db8-pmvlg           0/1     CrashLoopBackOff   979 (44s ago)     3d23h
piml-xgboost-77d48f9db8-wckmk           0/1     CrashLoopBackOff   533 (59s ago)     2d1h
piml-xgboost-77d48f9db8-wz657           0/1     CrashLoopBackOff   712 (2m39s ago)   2d21h
</code></pre>
<p>Docker command from Dockerfile</p>
<pre><code>CMD sleep 5;/usr/bin/amqp-consume --url=$BROKER_URL -q $QUEUE -c 1 ./docker_script.py
</code></pre>

JackLidge
<p>Deployment is not suitable for your use case. A deployment is designed for services that run permanently e.g. for serving a rest service or a worker that register to a message queue (seems to be tight to your use case). When a container stops, as you noted, kubernetes will restart it, but if that happens more often it is considered to be in an errornous state.</p>
<p>You may have two options:</p>
<ol>
<li><p>redesign your app to not stop after it finished its work but listen again on the queue for new messages</p>
</li>
<li><p>switch from deployment to cron job that runs every 5 seconds (and remove the sleep time from the container's command)</p>
</li>
</ol>
