<p>I am trying to achieve zero-downtime deployment on k8s. My deployment has one replica. The pod probes look like this:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: ${KUBE_NAMESPACE}
spec:
  selector:
    matchLabels:
      app: app
  replicas: 1
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
        - name: app-container
          imagePullPolicy: IfNotPresent
          image: ${DOCKER_IMAGE}:${IMAGE_TAG}
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
      terminationGracePeriodSeconds: 130
</code></pre>
<p>However, every time after the <code>kubectl rollout status</code> returns and reports rollout finished. I experience a small time of <code>bad gateway</code>.</p>
<p>Then I add a test that I let <code>/health</code> return 500 in <code>prestop</code> and wait at lease 20 seconds before actually stop the pod.</p>
<pre><code># If the app test the /tmp/prestop file exists, it will return 500.
          lifecycle:
            preStop:
              exec:
                command: [&quot;/bin/bash&quot;, &quot;-c&quot;, &quot;touch /tmp/prestop &amp;&amp; sleep 20&quot;]
</code></pre>
<p>Then I found after the k8s stop the pod, the traffic can still flow to the old pod(If I visit the /health, I can get a 500 result).</p>
<p>So it looks like the load balancer decides which pods can be used solely by the probe result. Since the probes have a period time, there is always a small window in which the pod stopped but the load balancer still doesn't know and can direct traffic to it, hence the user experiences downtime.</p>
<p>So my question is: in order to have a zero-downtime deployment, it seems a must to let the probes know the pod is stopping before actually stop the pod. Is this right? or I'm doing something wrong?</p>

kkpattern
<p>After digging around Google and doing some tests. I found it's not needed to manually replying 500 to probes after prestop.</p>
<p>According to <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">documentation</a></p>
<blockquote>
<p>At the same time as the kubelet is starting graceful shutdown, the control plane removes that shutting-down Pod from Endpoints (and, if enabled, EndpointSlice) objects where these represent a Service with a configured selector. ReplicaSets and other workload resources no longer treat the shutting-down Pod as a valid, in-service replica. Pods that shut down slowly cannot continue to serve traffic as load balancers (like the service proxy) remove the Pod from the list of endpoints as soon as the termination grace period begins.</p>
</blockquote>
<p>The pod won't get traffics after starting shutdown. But I also found this <a href="https://github.com/kubernetes/kubernetes/issues/47597" rel="nofollow noreferrer">issue</a> said there was indeed a delay between starting shutdown a pod to actually removing it from endpoints.</p>
<p>So instead of return 500 to probes in prestop, I simply sleep 60 seconds in prestop. At the sametime let the /health check return 200 with a status telling the node is in running or prestop status. Then I did a rollout and got following result:</p>
<pre class="lang-sh prettyprint-override"><code>b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717529.114602
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717530.59488
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717532.094305
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717533.5859041
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717535.086944
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717536.757241
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;running&quot;}' at 1612717538.57626
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;prestop&quot;}' at 1612717540.3773062
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;prestop&quot;}' at 1612717543.2204192
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;prestop&quot;}' at 1612717544.7196548
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;prestop&quot;}' at 1612717546.550169
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;prestop&quot;}' at 1612717548.01408
b'{&quot;node_id&quot;:&quot;a5c387f5df30&quot;,&quot;node_start_at&quot;:1612706851,&quot;status&quot;:&quot;prestop&quot;}' at 1612717549.471266
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717551.387528
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717553.49984
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717555.404394
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717558.1528351
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717559.64011
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717561.294955
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717563.366436
b'{&quot;node_id&quot;:&quot;17733ca118f4&quot;,&quot;node_start_at&quot;:1612717537,&quot;status&quot;:&quot;running&quot;}' at 1612717564.972768
</code></pre>
<p>The a5c387f5df30 node still got traffic after the prestop hook been called. After around 10 seconds, it never received traffic then. So it's not related to anything I did in prestop, it's purely a delay.</p>
<p>I did this test on AWS EKS with fargate. I don't know what's the status about other k8s platform.</p>
