<p>In a multiple node cluster we want to expose a service handling UDP traffic. There are two requirements:</p>
<ol>
<li>We want the service to be backed up by multiple pods (possibly running on different nodes) in order to scale horizontally.</li>
<li>The service needs the UDP source IP address of the client (i.e., should use DNAT instead of SNAT)</li>
</ol>
<p>Is that possible?</p>
<p>We currently use a <code>NodePort</code> service with <code>externalTrafficPolicy: local</code>. This forces DNAT but only the pod running on the requested node is receiving the traffic.
There doesn't seem to be a way to spread the load over multiple pods on multiple mnodes.</p>
<p>I already looked at <a href="https://kubernetes.io/docs/tutorials/services/source-ip/" rel="nofollow noreferrer">this Kubernetes tutorial</a> and also this article <a href="https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/" rel="nofollow noreferrer">here</a>.</p>

<p><strong>The Problem</strong></p>
<p>I feel like there is a need for some explanation before facing the actual issue(s) in order to understand <em>why</em> things do not work as expected:</p>
<p>Usually what happens when using <code>NodePort</code> is that you expose a port on every node in your cluster. When making a call to <code>node1:port</code> the traffic will then (same as with a <code>ClusterIP</code> type) be forwarded to one Pod that matches the <code>selector</code>, regardless of that Pod being on <code>node1</code> or another node.</p>
<p>Now comes the tricky part.
When using <code>externalTrafficPolicy: Local</code>, packages that arrive on a node that does not have a Pod on it will be dropped.
Perhaps the following illustration explains the behavior in a more understandable way.</p>
<p><code>NodePort</code> with default <code>externalTrafficPolicy: Cluster</code>:</p>
<pre><code>package --&gt; node1 --&gt; forwards to random pod on any node (node1 OR node2 OR ... nodeX)
</code></pre>
<p><code>NodePort</code> with <code>externalTrafficPolicy: Local</code>:</p>
<pre><code>package --&gt; node1 --&gt; forwards to pod on node1 (if pod exists on node1)

package --&gt; node1 --&gt; drops package (if there is no pod on node1)
</code></pre>
<p>So in essence to be able to properly distribute the load when using <code>externalTrafficPolicy: Local</code> two main issues need to be addressed:</p>
<ol>
<li>There has to be a Pod running on every node in order for packages not to be dropped</li>
<li>The client has to send packages to multiple nodes in order for the load to be distributed</li>
</ol>
<hr />
<p><strong>The solution</strong></p>
<p>The first issue can be resolved rather easily by using a <code>DaemonSet</code>. It will ensure that one instance of the Pod runs on every node in the cluster.</p>
<p>Alternatively one could also use a simple <code>Deployment</code>, manage the <code>replicas</code> manually and ensure proper distribution across the nodes <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">by using <code>podAntiAffinity</code></a>. This approach would take more effort to maintain since <code>replicas</code> must be adjusted manually but can be useful if you want to have more than just 1 Pod on each node.</p>
<p>Now for the second issue.
The easiest solution would be to let the client implement logic on his part and send requests to all the nodes in a round robin principle, however, that is not a very practical and/or realistic way of doing it.</p>
<p>Usually when using <code>NodePort</code> there is still a load balancer of some kind in front of it to distribute the load (not taking about the Kubernetes service type <code>LoadBalancer</code> here). This may seem redundant since by default <code>NodePort</code> will distribute the traffic across all the Pods anyways, however, the node that gets requested still gets the traffic and then another hop happens. Furthermore if only the same node is addressed at all time, once that node goes down (for whatever reason) traffic will never reach any of the Pods anyways. So for those (and many other reasons) a load balancer should <em>always</em> be used in combination with <code>NodePort</code>. To solve the issue simply configure the load balancer to preserve the source IP of the original client.</p>
<p>Furthermore, depending on what cloud you are running on, there is a chance of you being able to configure a service type <code>LoadBalancer</code> instead of <code>NodePort</code> (which basically is a <code>NodePort</code> service + a load balancer in front of it as described above) , configure it with <code>externalTrafficPolicy: Local</code> and address the first issue as described earlier and you achieved what you wanted to do.</p>
