<p>In Kubernetes, we have ClusterIp/Nodeport/LoadBalancer as the <strong>service</strong> to expose pods.
When there are <strong>multiple</strong> endpoints binds to one serivce (like deployment), then what is the policy Kubernetes route the traffic to one of the endpoints? Will it always try to respect a <code>load balancing</code> policy, or randomly selection?</p>

<p>Kubernetes uses <a href="https://en.wikipedia.org/wiki/Iptables" rel="noreferrer">iptables</a> to distribute traffic across a set of pods, as officially <a href="https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" rel="noreferrer">explained by kubernetes.io</a>. Basically what happens is when you create a <code>kind: service</code> object, K8s creates a virtual ClusterIP and instructs the kube-proxy daemonset to update iptables on each node so that requests matching that virtual IP will get load balanced across a set of pod IPs. The word "virtual" here means that ClusterIPs, unlike pod IPs, are not real IP addresses allocated by a network interface, and are merely used as a "filter" to match traffic and forward them to the right destination.</p>

<p>Kubernetes documentation says the load balancing method by default is round robin, but this is not entirely accurate. If you look at iptables on any of the worker nodes, you can see that for a given service <code>foo</code> with ClusterIP of 172.20.86.5 and 3 pods, the [overly simplified] iptables rules look like this:</p>

<pre><code>$ kubectl get service foo

NAME      TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
foo       ClusterIP  172.20.86.5   &lt;none&gt;        443:30937/TCP   12m
</code></pre>

<pre><code>Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-SVC-4NIQ26WEGJLLPEYD  tcp  --  anywhere             172.20.86.5          /* default/foo:https cluster IP */ tcp dpt:https
</code></pre>

<p>This <code>KUBE-SERVICES</code> chain rule looks for all traffic whose <code>destination</code> is 172.20.86.5, and applies rules defined in another chain called <code>KUBE-SVC-4NIQ26WEGJLLPEYD</code>:</p>

<pre><code>Chain KUBE-SVC-4NIQ26WEGJLLPEYD (2 references)
target     prot opt source               destination         
KUBE-SEP-4GQBH7D5EV5ANHLR  all  --  anywhere             anywhere             /* default/foo:https */ statistic mode random probability 0.33332999982
KUBE-SEP-XMNJYETXA5COSMOZ  all  --  anywhere             anywhere             /* default/foo:https */ statistic mode random probability 0.50000000000
KUBE-SEP-YGQ22DTWGVO4D4MM  all  --  anywhere             anywhere             /* default/foo:https */
</code></pre>

<p>This chain uses <code>statistic mode random probability</code> to randomly send traffic to one of the three chains defined (since I have three pods, I have three chains here each with 33.3% chance of being chosen to receive traffic). Each one of these chains is the final rule in sending the traffic to the backend pod IP. For example looking at the first one:</p>

<pre><code>Chain KUBE-SEP-4GQBH7D5EV5ANHLR (1 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             /* default/foo:https */ tcp to:10.100.1.164:12345
</code></pre>

<p>the <code>DNAT</code> directive forwards packets to IP address 10.100.1.164 (real pod IP) and port 12345 (which is what <code>foo</code> listens on). The other two chains (<code>KUBE-SEP-XMNJYETXA5COSMOZ</code> and <code>KUBE-SEP-YGQ22DTWGVO4D4MM</code>) are similar except each will have a different IP address.</p>

<p>Similarly, if your service type is <code>NodePort</code>, Kubernetes assigns a random port (from 30000-32767 by default) on the node. What's interesting here is that there is no process on the worker node actively listening on this port - instead, this is yet another iptables rule to match traffic and send it to the right set of pods:</p>

<pre><code>Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         
KUBE-SVC-4NIQ26WEGJLLPEYD  tcp  --  anywhere             anywhere             /* default/foo:https */ tcp dpt:30937
</code></pre>

<p>This rule matches inbound traffic going to port 30937 (<code>tcp dpt:30937</code>), and forwards it to chain <code>KUBE-SVC-4NIQ26WEGJLLPEYD</code>. But guess what: <code>KUBE-SVC-4NIQ26WEGJLLPEYD</code> is the same exact chain that cluster ip 172.20.86.5 matches on and sends traffic to, as shown above.</p>
