<p>We run a Kubernetes-compatible (OKD 3.11) on-prem / private cloud cluster with backend apps communicating with low-latency Redis databases used as caches and K/V stores. The new architecture design is about to divide worker nodes equally between two geographically distributed data centers (&quot;regions&quot;). We can assume static pairing between node names and regions, an now we have added labeling of nodes with region names as well.</p>
<p>What would be the recommended approach to protect low-latency communication with the in-memory databases, making client apps stick to the same region as the database they are allowed to use? Spinning up additional replicas of the databases is feasible, but does not prevent round-robin routing between the two regions...</p>
<p>Related: <a href="https://stackoverflow.com/questions/55743299/kubernetes-node-different-region-in-single-cluster">Kubernetes node different region in single cluster</a></p>

mirekphd
<p>Posting this out of comments as community wiki for better visibility, feel free to edit and expand.</p>
<hr />
<p>Best option to solve this question is to use <a href="https://istio.io/latest/docs/tasks/traffic-management/locality-load-balancing/" rel="nofollow noreferrer"><code>istio - Locality Load Balancing</code></a>. Major points from the link:</p>
<blockquote>
<p>A locality defines the geographic location of a workload instance
within your mesh. The following triplet defines a locality:</p>
<ul>
<li><p>Region: Represents a large geographic area, such as us-east. A region typically contains a number of availability zones. In
Kubernetes, the label topology.kubernetes.io/region determines a
node’s region.</p>
</li>
<li><p>Zone: A set of compute resources within a region. By running services in multiple zones within a region, failover can occur between
zones within the region while maintaining data locality with the
end-user. In Kubernetes, the label topology.kubernetes.io/zone
determines a node’s zone.</p>
</li>
<li><p>Sub-zone: Allows administrators to further subdivide zones for more fine-grained control, such as “same rack”. The sub-zone concept
doesn’t exist in Kubernetes. As a result, Istio introduced the custom
node label topology.istio.io/subzone to define a sub-zone.</p>
</li>
</ul>
<p>That means that a pod running in zone <code>bar</code> of region <code>foo</code> is not
considered to be local to a pod running in zone <code>bar</code> of region <code>baz</code>.</p>
</blockquote>
<hr />
<p>Another option that can be considered with traffic balancing adjusting is suggested in comments:</p>
<p>use <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity" rel="nofollow noreferrer"><code>nodeAffinity</code></a> to achieve consistency between scheduling <code>pods</code> and <code>nodes</code> in specific &quot;regions&quot;.</p>
<blockquote>
<p>There are currently two types of node affinity, called
<code>requiredDuringSchedulingIgnoredDuringExecution</code> and
<code>preferredDuringSchedulingIgnoredDuringExecution</code>. You can think of
them as &quot;hard&quot; and &quot;soft&quot; respectively, in the sense that the former
specifies rules that must be met for a pod to be scheduled onto a node
(similar to nodeSelector but using a more expressive syntax), while
the latter specifies preferences that the scheduler will try to
enforce but will not guarantee. The &quot;IgnoredDuringExecution&quot; part of
the names means that, similar to how nodeSelector works, if labels on
a node change at runtime such that the affinity rules on a pod are no
longer met, the pod continues to run on the node. In the future we
plan to offer <code>requiredDuringSchedulingRequiredDuringExecution</code> which
will be identical to <code>requiredDuringSchedulingIgnoredDuringExecution</code>
except that it will evict pods from nodes that cease to satisfy the
pods' node affinity requirements.</p>
<p>Thus an example of <code>requiredDuringSchedulingIgnoredDuringExecution</code>
would be &quot;only run the pod on nodes with Intel CPUs&quot; and an example
<code>preferredDuringSchedulingIgnoredDuringExecution</code> would be &quot;try to run
this set of pods in failure zone XYZ, but if it's not possible, then
allow some to run elsewhere&quot;.</p>
</blockquote>
<p><strong>Update</strong>: based on <a href="https://stackoverflow.com/questions/70006961/best-method-to-keep-client-server-traffic-in-the-same-region-node-in-kubernete#comment123810994_70036319">@mirekphd comment</a>, it will still not be fully functioning in a way it was asked to:</p>
<blockquote>
<p>It turns out that in practice Kubernetes does not really let us switch
off secondary zone, as soon as we spin up a realistic number of pod
replicas (just a few is enough to see it)... they keep at least some
pods in the other zone/DC/region by design (which is clever when you
realize that it removes the dependency on the docker registry
survival, at least under default imagePullPolicy for tagged images),
<a href="https://github.com/kubernetes/kubernetes/issues/99630#issuecomment-790740081" rel="nofollow noreferrer">GibHub issue #99630 - NodeAffinity preferredDuringSchedulingIgnoredDuringExecution doesn't work well</a></p>
</blockquote>
<p>Please refer to <a href="https://stackoverflow.com/a/70041931/15537201">@mirekphd's answer</a></p>
