<p>There seems to be two contradictory explanations of how NodePort services route traffic. Services can route traffic to one of the two, not both:</p>

<ol>
<li><strong>Nodes (through the kube-proxy)</strong> According to <code>kubectl explain Service.spec.externalTrafficPolicy</code> and <a href="https://www.asykim.com/blog/deep-dive-into-kubernetes-external-traffic-policies" rel="nofollow noreferrer">this article</a> that adds more detail, packets incoming to NodePort services with <code>Service.spec.externalTrafficPolicy=Local</code> set get routed to a kube-proxy, which then routes the packets to the corresponding pods its running.

<ul>
<li>This <a href="https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" rel="nofollow noreferrer">kube-proxy networking documentation</a> further supports this theory adding that endpoints add a rule in the service's IPtable that forwards traffic to nodes through the kube-proxy.</li>
</ul></li>
<li><strong>Pods</strong>: services update their IPtables from <code>endpoints</code>, which contain the IP addresses for the <em>pods</em> they can route to. Furthermore, if you remove your service's label selectors and <a href="https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/" rel="nofollow noreferrer">edit endpoints</a> you can change where your traffic is routed to.</li>
</ol>

<p>If one of these is right, then I must be misunderstanding something.</p>

<ul>
<li>If services route to <strong>nodes</strong>, then why can I edit <code>endpoints</code> without breaking the IPtables? </li>
<li>If services route to <strong>pods</strong>, then why would services go through the trouble of routing to nodes when <code>Service.spec.externalTrafficPolicy</code> is set?</li>
</ul>

<p>A <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noreferrer">Service</a> is a virtual address/port managed by <code>kube-proxy</code>. Services forward traffic to their associated endpoints, which are usually pods but as you mentioned, can be set to any destination IP/Port. </p>

<p>A <a href="https://kubernetes.io/docs/concepts/services-networking/service/#nodeport" rel="noreferrer">NodePort Service</a> doesn't change the endpoint side of the service, the NodePort allows external traffic into Service via a port on a node.</p>

<h3>Breakdown of a Service</h3>

<p><code>kube-proxy</code> can use <a href="https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" rel="noreferrer">3 methods</a> to implement the forwarding of a service from Node to destination. </p>

<ul>
<li>a user proxy</li>
<li>iptables</li>
<li>ipvs</li>
</ul>

<p>Most clusters use iptables, which is what is described below. I use the term "forward" instead of "route" because services use <a href="https://en.wikipedia.org/wiki/Network_address_translation" rel="noreferrer">Network Address Translation</a> (or the proxy) to "forward" traffic rather than standard network routing. </p>

<p>The service <code>ClusterIP</code> is a virtual entity managed by <code>kube-proxy</code>. This address/port combination is available on every node in the cluster and forwards any local (pod) service traffic to the endpoints IP and port. </p>

<pre><code>                                         / Pod (remote node)
Pod -- ClusterIP/Port -- KUBE-SVC-NAT  --  Pod
                                         \ Pod (remote node)
</code></pre>

<p>A service with a <code>NodePort</code> is the same as above, with the addition of a way to forward external traffic into the cluster via a Node. <code>kube-proxy</code> manages an additional rule to watch for external traffic and forward it into the same service rules. </p>

<pre><code>Ext --     NodePort   \                / Pod (remote node)
                        KUBE-SVC-NAT --  Pod
Pod -- ClusterIP/Port /                \ Pod (remote node)
</code></pre>

<p>The <code>externalTrafficPolicy=Local</code> setting makes a NodePort service use <em>only</em> a local Pod to service the incoming traffic. This avoids a network hop which removes the need to rewrite the source of the packet (via NAT). This results in the real network IP arriving at the pod servicing the connection, rather than one of the cluster nodes being the source IP. </p>

<pre><code>Ext --     NodePort   \                  Pod (remote node)
                        KUBE-SVC-NAT --  Pod (local)
Pod -- ClusterIP/Port /                  Pod (remote node)
</code></pre>

<h3>iptables</h3>

<p>I recommend attempting to trace a connection from source to destination for a service or nodeport on a host. It requires a bit of iptables knowledge but I think it's worthwhile</p>

<p>To list all the services ip/ports that will be forwarded: </p>

<p><code>iptables -vnL -t nat KUBE-SERVICES</code></p>

<p>To list all the nodeports that will be forwarded:</p>

<p><code>iptables -vnL -t nat KUBE-NODEPORTS</code></p>

<p>Once you have the rule you can jump through <code>KUBE-SVC-XXX</code> "target" rules in the full output. </p>

<p><code>iptables -vnL -t nat | less</code></p>
