<p>I am trying to achieve zero-downtime deployment using Kubernetes. But every time I do the upgrade of the deployment using a new image, I am seeing 2-3 seconds of downtime. I am testing this using a Hello-World sort of application but still could not achieve it. I am deploying my application using the Helm charts.</p>

<p>Following the online blogs and resources, I am using Readiness-Probe and Rolling Update strategy in my Deployment.yaml file. But this gives me no success.
I have created a <code>/health</code> end-point which simply returns <code>200</code> status code as a check for readiness probe. I expected that after using readiness probes and RollingUpdate strategy in Kubernetes I would be able to achieve zero-downtime of my service when I upgrade the image of the container. The request to my service goes through an Amazon ELB.</p>

<p>Deployment.yaml file is as below:</p>

<pre><code>apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: wine-deployment
  labels:
    app: wine-store
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: wine-store
  replicas: 2
  template:
    metadata:
      labels:
        app: wine-store
    spec:
      containers:
        - name: {{ .Chart.Name }}
          resources:
            limits:
              cpu: 250m
            requests:
              cpu: 200m
          image: "my-private-image-repository-with-tag-and-version-goes-here-which-i-have-hidden-here"
          imagePullPolicy: Always
          env:
          - name: GET_HOSTS_FROM
            value: dns
          ports:
          - containerPort: 8089
            name: testing-port
          readinessProbe:
            httpGet:
              path: /health
              port: 8089
            initialDelaySeconds: 3
            periodSeconds: 3 
</code></pre>

<p>Service.yaml file:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: wine-service
  labels:
    app: wine-store
spec:
  ports:
    - port: 80
      targetPort: 8089
      protocol: TCP
  selector:
    app: wine-store
</code></pre>

<p>Ingress.yaml file:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: wine-ingress
  annotations:
     kubernetes.io/ingress.class: public-nginx
spec:
  rules:
    - host: my-service-my-internal-domain.com
      http:
        paths:
          - path: /
            backend:
              serviceName: wine-service
              servicePort: 80
</code></pre>

<p>I expect the downtime to be zero when I am upgrading the image using <code>helm upgrade</code> command. Meanwhile, when the upgrade is in progress, I continuously hit my service using a curl command. This curl command gives me <code>503-service Temporarily un-available</code> errors for 2-3 seconds and then again the service is up. I expect that this downtime does not happens. </p>

<p>This issue is caused by the Service VIP using iptables. You haven't done anything wrong - it's a limitation of current Kubernetes.</p>

<p>When the readiness probe on the new pod passes, the old pod is terminated and kube-proxy rewrites the iptables for the service. However, a request can hit the service after the old pod is terminated but before iptables has been updated resulting in a 503.</p>

<p>A simple workaround is to delay termination by using a <code>preStop</code> lifecycle hook:</p>

<pre><code>lifecycle:
  preStop:
    exec:
      command: ["/bin/bash", "-c", "sleep 10"]
</code></pre>

<p>It'd probably not relevant in this case, but implementing graceful termination in your application is a good idea. Intercept the TERM signal and wait for your application to finish handling any requests that it has already received rather than just exiting immediately.</p>

<p>Alternatively, more replicas, a low <code>maxUnavailable</code> and a high <code>maxSurge</code> will all reduce the probability of requests hitting a terminating pod.</p>

<p>For more info:
<a href="https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables" rel="noreferrer">https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables</a>
<a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods" rel="noreferrer">https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods</a></p>

<p>Another answer mistakenly suggests you need a liveness probe. While it's a good idea to have a liveness probe, it won't effect the issue that you are experiencing. With no liveness probe defined the default state is Success.</p>

<p>In the context of a rolling deployment a liveness probe will be irrelevant - Once the readiness probe on the new pod passes the old pod will be sent the TERM signal and iptables will be updated. Now that the old pod is terminating, any liveness probe is irrelevant as its only function is to cause a pod to be restarted if the liveness probe fails.</p>

<p>Any liveness probe on the new pod again is irrelevant. When the pod is first started it is considered live by default. Only after the <code>initialDelaySeconds</code> of the liveness probe would it start being checked and, if it failed, the pod would be terminated.</p>

<p><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes" rel="noreferrer">https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes</a></p>
