<p>I'm in the process of packaging an application to deploy and run on an on-prem Kubernetes cluster. Everything has been great so far, but I'm struggling on a component that is essentially a database. This component requires 3 master nodes, with 2 replicas. What's interesting is the master node itself counts as a replica, and needs to be configured as such (this is for sharding purposes). Since this is on-prem, I need to balance the master node and replica pairs across 3 physical hosts, with unique node and replica pairs running on each host. </p>

<p>This is an example of what the layout would look like, where N == a master node, and R == replica. </p>

<pre><code>Host1  Host2  Host3
 N1R1   N2R1   N3R1
 N3R2   N1R2   N2R2
</code></pre>

<p>I can generate the required config using StatefulSets and some helm magic with the init container ordinals, but I'm struggling to pin the N1R1 and N3R2 to Host 1, N2R1 and N1R2 instances to Host 2, and N3R1 and N2R2 instances to Host 3. I need to do this to ensure data integrity. Am I missing something with persistent volume controllers? Or some setting in StatefulSets that lets me access the underlying Kubernetes node to define which node and replica pairs run where?</p>

<p>What about Affinity and AntiAffinity? Can I leverage these to force a secondary replica instance to never be on the same node as it's master instance?</p>

user797963
<p>You are looking for Anti-Affinity. To avoid scheduling 2 Pods on the same Node you need to use <code>topologyKey: "kubernetes.io/hostname"</code> in the resource description.</p>

<p>Here is the <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#never-co-located-in-the-same-node" rel="nofollow noreferrer">link</a> on the official documentation with the example.</p>

<p>And here is the <a href="https://livebook.manning.com/#!/book/kubernetes-in-action/chapter-16" rel="nofollow noreferrer">link</a> for deeper understanding how it works.</p>
