<p>I set up a (very) simple deployment with GKE on a GKE Autopilot cluster running the latest version of Kubernetes (<code>1.18.15-gke.1501</code>) and attached an ingress (external HTTP(s) load balancer) that links to a simple <code>ClusterIP</code> service.</p>
<p>Whenever I update the deployment with a new image, I experience about 5-15 minutes of downtime where the load balancer returns a 502 error. It seems like the control plane creates the new, updated pod, allows the service-level health checks to go through (not the load-balancer ones, it doesn't create the NEG yet), then kills the older pod while at the same time setting up the new NEG. It then doesn't remove the old NEG until a variable amount of time later.</p>
<p><a href="https://i.stack.imgur.com/j3ZKz.png" rel="noreferrer"><img src="https://i.stack.imgur.com/j3ZKz.png" alt="enter image description here" /></a></p>
<p>Logs on the pods show that health checks are going through, but the GKE dashboard show inconsistent results for the Ingress' state. The ingress will show as fine, but the service will 502.</p>
<h2>Things I've tried</h2>
<ul>
<li>Increasing the number of pods from 1 to 3. This helped on some deploys, but on every other deploy it <em>increased</em> the amount of time it took for the load balancer to resolve correctly.</li>
<li>Attempted setting <code>maxSurge</code> to 1 and <code>maxUnavailable</code> to 0. This did not improve the downtime at all.</li>
<li>Adding <code>lifecycle.preStop.exec.command: [&quot;sleep&quot;, &quot;60&quot;]</code> to the container on the deployment. <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing#traffic_does_not_reach_endpoints" rel="noreferrer">This was suggested in the GKE docs here.</a></li>
<li>Recreating the ingress, service, deployments, and clusters multiple times.</li>
<li>Adding a <code>BackendConfig</code> to the service that adds slower draining on it.</li>
<li>Adding a readiness gate found in the docs that's supposed to fix this, but for some reason doesn't?</li>
</ul>
<p>None of the above have helped or made any noticeable difference in how long things were down.</p>
<p>I'm really, really confused by why this isn't working. It feels like I'm missing something <em>really</em> obvious, but this is also such a simple config that you'd think it'd... just work?? Anyone have any idea on what's going on?</p>
<h3>Config files</h3>
<p>Deployment config:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: foundation-deployment
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: foundation-web
  template:
    metadata:
      labels:
        app: foundation-web
    spec:
      readinessGates:
        - conditionType: &quot;cloud.google.com/load-balancer-neg-ready&quot;

      serviceAccountName: foundation-database-account
      containers:
        # Run Cloud SQL proxy so we can safely connect to Postgres on localhost.
        - name: cloud-sql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.17
          resources:
            requests:
              cpu: &quot;250m&quot;
              memory: 100Mi
            limits:
              cpu: &quot;500m&quot;
              memory: 100Mi
          command:
            - &quot;/cloud_sql_proxy&quot;
            - &quot;-instances=nine-foundation:us-central1:foundation-staging=tcp:5432&quot;
          securityContext:
            runAsNonRoot: true
        # Main container config
        - name: foundation-web
          image: gcr.io/project-name/foundation_web:latest
          imagePullPolicy: Always
          lifecycle:
            preStop:
              exec:
                command: [&quot;sleep&quot;, &quot;60&quot;]
          env:
           # Env variables
          resources:
            requests:
              memory: &quot;500Mi&quot;
              cpu: &quot;500m&quot;
            limits:
              memory: &quot;1000Mi&quot;
              cpu: &quot;1&quot;
          livenessProbe:
            httpGet:
              path: /healthz
              port: 4000
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /healthz
              port: 4000
            initialDelaySeconds: 10
            periodSeconds: 10
          ports:
            - containerPort: 4000
</code></pre>
<p>Service config:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: foundation-web-service
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
    cloud.google.com/backend-config: '{&quot;ports&quot;: {&quot;4000&quot;: &quot;foundation-service-config&quot;}}'
spec:
  type: ClusterIP
  selector:
    app: foundation-web
  ports:
    - port: 4000
      targetPort: 4000
</code></pre>
<p>BackendConfig:</p>
<pre><code>apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: foundation-service-config
spec:
  # sessionAffinity:
  #   affinityType: &quot;GENERATED_COOKIE&quot;
  #   affinityCookieTtlSec: 120
  connectionDraining:
    drainingTimeoutSec: 60
</code></pre>
<p>Ingress config:</p>
<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: foundation-web-ingress
  labels:
    name: foundation-web-ingress
spec:
  backend:
    serviceName: foundation-web-service
    servicePort: 4000
</code></pre>

<p>I think this might be to do with the cloud sql auth proxy sidecar not terminating properly, leading to the load balancer getting in a twist.</p>
<p>Note this from the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing#traffic_does_not_reach_endpoints" rel="nofollow noreferrer">GCP docs</a> (I've made the key parts <em>italics</em>)</p>
<blockquote>
<p><strong>Symptoms</strong></p>
<p>502 errors or rejected connections. Potential causes New endpoints
generally become reachable after attaching them to the load balancer,
provided that they respond to health checks. You might encounter 502
errors or rejected connections if traffic cannot reach the endpoints.</p>
<p><em>502 errors and rejected connections can also be caused by a container
that doesn't handle SIGTERM. If a container doesn't explicitly handle
SIGTERM, it immediately terminates and stops handling requests. The
load balancer continues to send incoming traffic to the terminated
container, leading to errors.</em></p>
<p>The container native load balancer only has one backend endpoint.
During a rolling update, the old endpoint gets deprogrammed before the
new endpoint gets programmed.</p>
<p>Backend Pod(s) are deployed into a new zone for the first time after a
container native load balancer is provisioned. Load balancer
infrastructure is programmed in a zone when there is at least one
endpoint in the zone. When a new endpoint is added to a zone, load
balancer infrastructure is programmed and causes service disruptions.</p>
<p><strong>Resolution</strong></p>
<p><em>Configure containers to handle SIGTERM and continue
responding to requests throughout the termination grace period (30
seconds by default)</em>. Configure Pods to begin failing health checks
when they receive SIGTERM. This signals the load balancer to stop
sending traffic to the Pod while endpoint deprograming is in progress.</p>
</blockquote>
<p>By default, the proxy doesn't handle SIGTERM nicely, and wont exit gracefully on a SIGTERM (see <a href="https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/128" rel="nofollow noreferrer">related issue</a>), however it now has a nice flag for handling this, so you can use something like</p>
<pre><code>- name: cloud-sql-proxy
      image: gcr.io/cloudsql-docker/gce-proxy:1.23.1
      command:
        - &quot;/cloud_sql_proxy&quot;
        - &quot;-instances={{ .Values.postgresConnectionName }}=tcp:5432&quot;
        - &quot;-term_timeout=60s&quot;
      securityContext:
        runAsNonRoot: true
      resources:
        requests:
          memory: &quot;2Gi&quot;
          cpu:    &quot;1&quot;
</code></pre>
<p>Adding the term_timeout flag mainly fixed it for me but was still seeing the occasional 502 during a deployment. Upping replicas to 3 (my cluster is regional, so I wanted to cover all the zones) seemed to help, once I had the term_timeout in place.</p>
