<p>We are using K8S in a managed Azure environment, Minikube in Ubuntu and a Rancher cluster built on on-prem machines and in general, our deployments take up to about 30 seconds to pull containers, run up and be ready. However, my latest attempt to create a deployment (on-prem) takes upwards of a minute and sometimes longer. It is a small web service which is very similar to our other deployments. The only (obvious) difference is the use of a startup probe and a liveness probe, although some of our other services do have probes, they are different though.</p>
<p>After removing Octopus deploy from the equation by extracting the yaml it was running and using kubectl, as soon as the (single) pod starts, I start reading the logs and as expected, the startup and liveness probes are called very quickly. Startup succeeds and the cluster starts calling the live probe, which also succeeds. However, if I use <code>kubectl describe</code> on the pod, it shows Initialized and PodScheduled as True but ContainersReady (there is one container) and Ready are both false for around a minute. I can't see what would cause this other than probe failures but these are logged as successful.</p>
<p>They eventually start and work OK but I don't know why they take so long.</p>
<pre class="lang-yaml prettyprint-override"><code>kind: Deployment 
apiVersion: apps/v1 
metadata: 
  name: 'redirect-files-deployments-28775' 
  labels: 
    Octopus.Kubernetes.SelectionStrategyVersion: &quot;SelectionStrategyVersion2&quot; 
    OtherOctopusLabels
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      Octopus.Kubernetes.DeploymentName: 'redirect-files-deployments-28775' 

  template: 
    metadata: 
      labels: 
        Octopus.Kubernetes.SelectionStrategyVersion: &quot;SelectionStrategyVersion2&quot; 
        OtherOctopusLabels

    spec: 
      containers: 
      - name: redirect-files 
        image: ourregistry.azurecr.io/microservices.redirectfiles:1.0.34 
        ports: 
        - name: http 
          containerPort: 80 
          protocol: TCP 
        env: 
        - removed connection strings etc
        livenessProbe: 
          httpGet: 
            path: /api/version 
            port: 80 
            scheme: HTTP 
          successThreshold: 1 
        startupProbe: 
          httpGet: 
            path: /healthcheck 
            port: 80 
            scheme: HTTP 
            httpHeaders: 
            - name: X-SS-Authorisation 
              value: asdkjlkwe098sad0akkrweklkrew 

          initialDelaySeconds: 5 
          timeoutSeconds: 5 
      imagePullSecrets: 
      - name: octopus-feedcred-feeds-azure-container-registry 
</code></pre>

Luke Briner
<p>So the cause was the startup and/or liveness probes. When I removed them, the deployment time went from over a minute to 18 seconds, despite the logs proving that the probes were called successfully very quickly after containers were started.</p>
<p>At least I now have something more concrete to look for.</p>
