<p>I have <a href="https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html" rel="nofollow noreferrer">HPA</a> for my Kubernetes-deployed app with <a href="https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html" rel="nofollow noreferrer">cluster autoscaler</a>.
Scaling works properly for both pods and nodes, but during production load spikes I see a lot of 502 errors from ALB (<a href="https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html" rel="nofollow noreferrer">aws-load-balancer-controller</a>).</p>
<p>It seems like I have enabled everything to achieve zero-downtime deployment / scaling:</p>
<ul>
<li>pod readiness probe is in place</li>
</ul>
<pre><code>       readinessProbe:
         httpGet:
           path: /_healthcheck/
           port: 80
</code></pre>
<ul>
<li>pod readiness gate <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/pod_readiness_gate/" rel="nofollow noreferrer">is enabled</a></li>
<li>ingress annotation uses <code>ip</code> target type</li>
</ul>
<pre><code>alb.ingress.kubernetes.io/target-type: ip
</code></pre>
<ul>
<li>healthcheck parameters are specified on the ingress resource</li>
</ul>
<pre><code>alb.ingress.kubernetes.io/healthcheck-path: &quot;/healthcheck/&quot;
alb.ingress.kubernetes.io/healthcheck-interval-seconds: &quot;10&quot;
</code></pre>
<p>but that doesn't help.</p>
<p>How to properly debug this kind of issue and which other parameters should I tune to completely eliminate 5xx errors from my load balancer?</p>

<p>Here's a list of some extra things that I've added to my configuration alongside those mentioned above</p>
<ul>
<li>container <code>preStop</code> <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks" rel="nofollow noreferrer">hook</a></li>
</ul>
<pre><code>lifecycle:
  preStop:
    exec:
      command: [&quot;/bin/sleep&quot;, &quot;30&quot;]
</code></pre>
<ul>
<li><p><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">termination grace period</a> on a pod <code>terminationGracePeriodSeconds: 40</code> (sleep time from the above + 10-15 seconds)</p>
</li>
<li><p>tune deregistration delay value on a target group by setting</p>
</li>
</ul>
<pre><code>alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
</code></pre>
<p>this annotation on an ingress resources. Usually the value should match your timeout on backend webserver (we don't want to have a target around more than it requires for the longest possible request to finish).</p>
<p><strong>The main idea</strong> behind this tuning is to make sure changes of the Pods state have enough time to propagate to the underlying AWS resources, so traffic is no longer routed from ALB to the pod within target group that has been already marked as terminated/unhealthy by k8s.</p>
<p>P.S. Make sure to always have enough pods to handle incoming requests (this is especially important for synchronous workers when doing rolling redeploy). <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment" rel="nofollow noreferrer">Consider</a> lower values for <code>maxUnavailable</code> and higher values for <code>maxSurge</code> in case your cluster/worker nodes have the capacity to allocate these extra pods. So if your pod handles 100 reqs/min on average on your load is 400 reqs/min make sure <code>num of replicas</code> - <code>maxUnavailable</code> &gt; 4 (total reqs / reqs per pod)</p>
