<p>We have two types of services that we run on AWS EKS:</p>

<ul>
<li>external-facing services which we expose through an application-level load balancer using aws-alb-ingress-controller</li>
<li>internal-facing services which we use both directly through the service name (for EKS applications) and through an internal application-level loadbalancer also using aws-alb-ingress-controller (for non-EKS applications)</li>
</ul>

<p>I would like to understand the performance implications of choosing Nodeport, ClusterIP or Headless Service for both the external and internal services. I have the setup working with all three options.</p>

<p>If I understanding the networking correctly, it seems that a Headless Service requires less hops and would hence be (slightly) faster? <a href="https://medium.com/faun/kubernetes-headless-service-vs-clusterip-and-traffic-distribution-904b058f0dfd" rel="nofollow noreferrer">This article</a> however seems to suggest that a Headless Service would not be properly load balanced when called directly. Is this correct? And would this still hold when called through the external (or internal) ALB?</p>

<p>Is there any difference in performance for NodePort vs ClusterIP?</p>

<p>Finally, what is the most elegant/performant way of using internal services from outside of the cluster (where we don't have access to the Kubernetes DNS) but within the same VPC? Would it be to use ClusterIp and specify the IP address in the service definition so it remains stable? Or are there better options?</p>

chrisvdb
<p>I've put more detailed info on the each of the connection forwarding types and how the services are forwarded down under the headings belowfor context to my answers. </p>

<blockquote>
  <p>If I understanding the networking correctly, it seems that a Headless Service requires less hops and would hence be (slightly) faster? </p>
</blockquote>

<p>Not substantially faster. The "extra hop" is the packet traversing local lookup tables which it traverses anyway so not a noticeable difference. The destination pod is still going to be the same number of actual network hops away. </p>

<p>If you have 1000's of services that run on a single pod and could be headless then you might use that to limit the number of iptables NAT rules and speed rule processing up (see iptables v ipvs below).</p>

<blockquote>
  <p>Is &lt; a headless service not load balanced > correct? And would this still hold when called through the external (or internal) ALB?</p>
</blockquote>

<p>Yes it is correct, the client (or ALB) would need to implement the load balancing across the Pod IP's.</p>

<blockquote>
  <p>Is there any difference in performance for NodePort vs ClusterIP?</p>
</blockquote>

<p>A NodePort has a possible extra network hop from the entry node to the node running the pod. Assuming the ClusterIP ranges are routed to the correct node (and routed at all)</p>

<p>If you happen to be using a service type: <code>LoadBalancer</code> this behaviour can change by setting [<code>.spec.externalTrafficPolicy</code> to <code>Local</code>][<a href="https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support]" rel="noreferrer">https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support]</a> which means traffic will only be directed to a local pod.</p>

<blockquote>
  <p>Finally, what is the most elegant/performant way of using internal services from outside of the cluster</p>
</blockquote>

<p>I would say use the <a href="https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html" rel="noreferrer">AWS ALB Ingress Controller</a> with the <code>alb.ingress.kubernetes.io/target-type: ip</code> annotation. The k8s config from the cluster will be pushed out to the ALB via the ingress controller and address pods directly without traversing any connection forwarding or extra hops. All cluster reconfig will be automatically pushed out. </p>

<p>There is a little bit of latency for config to get to the ALB compared to cluster <code>kube-proxy</code> reconfiguration. Something like a rolling deployment might not be as seamless as the updates arrive after a pod is gone. The ALB's are equipped to handle the outage themselves, eventually.</p>

<h2>Kubernetes Connection Forwarding</h2>

<p>There is a <code>kube-proxy</code> process running on each node which manages how and where connections are forwared. There are 3 options for how <code>kube-proxy</code> does that: <a href="https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" rel="noreferrer">Userspace proxy, iptables or IPVS</a>. Most clusters will be on iptables and that will cater for the vast majority of use cases.</p>

<h3><a href="https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace" rel="noreferrer">Userspace proxy</a></h3>

<p>The forwarding is via a process that runs in userspace to terminate and forward the connections. It's slow. It's unlikely you are using it, don't use it. </p>

<h3><a href="https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables" rel="noreferrer">iptables</a></h3>

<p>iptables forwards connections in kernel via NAT, which is fast. This is most common setup and will cover 90% of use cases. New connections are shared evenly between all nodes running pods for a service. </p>

<h3><a href="https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs" rel="noreferrer">IPVS</a></h3>

<p>Runs in kernel, it is fast and scalable. If you shift a traffic to a large number of apps this might improve the forwarding performance. It also supports different service load balancing modes:</p>

<pre><code>- rr: round-robin
- lc: least connection (smallest number of open connections)
- dh: destination hashing
- sh: source hashing
- sed: shortest expected delay
- nq: never queue
</code></pre>

<h2>Access to services</h2>

<p>My explanations are iptables based as I haven't done much detailed work with ipvs clusters yet.  I'm gonna handwave the ipvs complexity away and say it's basically the same as iptables, just with faster rule processing as the number of rules increases on huge clusters (i.e number of pods/services/network policies). </p>

<p>I'm also ignoring the userspace proxy in the description, due to the overhead just don't use it.</p>

<p>The basic thing to understand is a "Service ClusterIP" is a virtual construct in the cluster that only exists as rule for where the traffic should go. Every node maintains this rule mapping of all ClusterIP/port to PodIP/port (via <code>kube-proxy</code>)</p>

<h3>Nodeport</h3>

<p>ALB routes to any node, The node/nodeport forwards the connection to a pod handling the service. This could be a remote pod which would involve sending traffic back out over the "wire".</p>

<p>ALB > wire > Node > Kernel Forward to SVC ( > wire if remote node ) > Pod</p>

<h3>ClusterIP</h3>

<p>Using the ClusterIP for direct access depends on the Service cluster IP ranges being routed to the correct node. Sometimes they aren't routed at all.</p>

<p>ALB > wire > Node > Kernel Forward to SVC > Pod</p>

<p>The "Kernel Forward to SVC" step can be skipped with an ALB annotation without using a headless service. </p>

<h3>Headless Service</h3>

<p>Again, Pod IP's aren't always addressable from outside the cluster depending on the network setup. You should be fine on EKS.</p>

<p>ALB > wire > Node > Pod</p>

<h3>Note</h3>

<p>I'll suffix this with requests are probably looking at &lt; 1ms of additional latency if a connection is forwarded to a node in a VPC. Enhanced networking instances at the low end of that. Inter availability-zone comms might be a tad higher than intra-AZ. If you happened to have a geographically separated cluster it might increase the importance of controlling traffic flow. For example having a tunnelled calico network that actually jumped over a number of real networks. </p>
