<p>We have a situation now when 2 pods of same type can run on the same node. Sometimes during restarts and rescheduling 2 pods happen to be on the same node and when this node itself is rescheduled all of our pods are gone for a while resulting in connection troubles (we have just 2 pods that are load balanced).</p>

<p>I think the best way to fix it is to not allow 2 pods to run on the same node and to use inter-pod anti-affinity for that. </p>

<p>Is this the correct solution to the problem? I tried to understand it but got a bit bogged down with <code>topologyKey</code> and the syntax. Can someone explain/give an example on how to achieve this?</p>

<p>Yes, you are right Affinity is your friend here and is the correct solution.</p>
<p>Node Affinity : will help your app or microservice to stick to a particular kind of node (in a multinode architecture) like below my app <code>ngnix-ms</code> always sticks to the nodes which have a label <code>role=admin</code>.</p>
<p>The pod antiAffinity rule: sticks to the node labels (topologyKey) and makes sure that group (marked with topologyKey) of nodes.</p>
<p>If there is a node which already has a pod with the label <code>component=nginx</code>, K8s won't allow to spin up a pod.</p>
<p>Here is the explanation:</p>
<pre><code>affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: role
              operator: In
              values:
              - app-1
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: component
                operator: In
                values:
                - nginx-ms
          topologyKey: &quot;kubernetes.io/1-hostname&quot;
</code></pre>
<p>and</p>
<pre><code>kubectl get node --show-labels
    
NAME                  STATUS   ROLES    AGE   VERSION   LABELS
xx-admin-1      Ready    master   19d   v1.13.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/1-hostname=xx-admin-1,node-role.kubernetes.io/master=,role=admin
xx-admin-2      Ready    master   19d   v1.13.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/1-hostname=xx-admin-2,node-role.kubernetes.io/master=,role=admin
xx-plat-1-1     Ready    &lt;none&gt;   19d   v1.13.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/2-hostname=xx-plat-1-1,role=admin
xx-plat-2-1     Ready    &lt;none&gt;   19d   v1.13.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/2-hostname=xx-plat-2-1,role=admin
</code></pre>
<p>Explanation of topologyKey:
Think of it as a label, now you can have two different topologies in the same cluster.</p>
<p>example: <code>kubernetes.io/1-hostname</code> and <code>kubernetes.io/2-hostname</code></p>
<p>Now when you are doing podAntiAffinity you set `topologyKey:</p>
<p><code>kubernetes.io/1-hostname</code></p>
<p>Then your rule is valid in all the nodes with that topologyKey but your rule is not valid in <code>topologyKey: kubernetes.io/2-hostname</code> labelled nodes.</p>
<p>Hence, in my example pods are scheduled within nodes with the label <code>kubernetes.io/1-hostname</code> and has podAntiAffinity implied, but nodes with labels <code>kubernetes.io/2-hostname</code> doesn't have the podAntiAffinity rule!</p>
