<p>I have simple web application deployed to Kubernetes cluster (EKS) with <a href="https://github.com/kubernetes-sigs/aws-load-balancer-controller" rel="nofollow noreferrer">aws load balancer controller</a> ingress</p>
<p><a href="https://i.stack.imgur.com/MmFF9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MmFF9.png" alt="schema" /></a></p>
<p>When accessing app in the intended way on ALB endpoint the performance is very poor (2-3x worse than regular deployment on bare metal instance). Benchmark was done with <a href="https://github.com/rakyll/hey" rel="nofollow noreferrer">Hey</a></p>
<pre><code>$ hey -t 30 -z 1m https://k8s-default-ingre-fdeb4c8b98-1975505070.us-east-1.elb.amazonaws.com/
# 5-10 reqs/s
$ hey -t 30 -z 1m http://172.16.3.37/  # from another pod accessing directly by its IP
# 20-30 reqs/s 
</code></pre>
<p>If accessed from same/another pod or from different instance(node) when exposed as a <code>NodePort</code> performance stays the same, so I'm assuming something wrong with ingress/ALB.</p>
<p>How to identify bottleneck and debug such kind of an issue?</p>
<p>Here's my config</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/healthcheck-path: &quot;/healthz/&quot;
    alb.ingress.kubernetes.io/certificate-arn:  &quot;arn:aws:acm:us-east-1::certificate/cert&quot;
    alb.ingress.kubernetes.io/listen-ports: '[{&quot;HTTP&quot;: 80}, {&quot;HTTPS&quot;:443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{&quot;Type&quot;: &quot;redirect&quot;, &quot;RedirectConfig&quot;: { &quot;Protocol&quot;: &quot;HTTPS&quot;, &quot;Port&quot;: &quot;443&quot;, &quot;StatusCode&quot;: &quot;HTTP_301&quot;}}'
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ssl-redirect
            port:
              name: use-annotation
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80
</code></pre>

Most Wanted
<ol>
<li>Try and run kubernetes networking benchmark on both the clusters (bare-metal and EKS) -
<a href="https://github.com/InfraBuilder/k8s-bench-suite" rel="nofollow noreferrer">https://github.com/InfraBuilder/k8s-bench-suite</a></li>
</ol>
<p>This shows how pod2pod and pod2svc latencies are on both clusters. If drastically different, it must probably be the underlying network -</p>
<ol start="2">
<li>Run iperf3 benchmark between nodes -
<a href="https://aws.amazon.com/premiumsupport/knowledge-center/network-throughput-benchmark-linux-ec2/" rel="nofollow noreferrer">https://aws.amazon.com/premiumsupport/knowledge-center/network-throughput-benchmark-linux-ec2/</a></li>
</ol>
<p>This would probably give you an idea of how nodes separated far apart on EKS cluster impact latencies, uneven networking bandwidth, etc.</p>
<p>If the latencies are somewhat similar, you might want to dig into your application code using something like jaeger or application profiler to get breakdown of latencies.</p>
