<p>So I have 4 nodes. 1 is System, 1 is Dev, 1 is Qa and 1 is UAT.</p>
<p>My affinity is as follows:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth
  namespace: dev
  labels:
    app: auth
    environment: dev
    app-role: api
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auth
  template:
    metadata:
      labels:
        app: auth
        environment: dev
        app-role: api
        tier: backend
      annotations:
        build: _{Tag}_
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - auth
            topologyKey: kubernetes.io/hostname
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: environment
                operator: In
                values:
                - dev
      containers:
        - name: companyauth
          image: company.azurecr.io/auth:_{Tag}_
          imagePullPolicy: Always
          env:
            - name: ConnectionStrings__DevAuth
              value: dev
          ports:
            - containerPort: 80
      imagePullSecrets:
        - name: ips
</code></pre>
<p>It is my intention to make sure that on my production cluster, which has 3 nodes in 3 different availability zones. That all the pods will be scheduled on a different node/availability zone. However, it appears that if I already have pods scheduled on a node, then when I do a deployment it will not overwrite the pods that already exist.</p>
<p>0/4 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match node selector.</p>
<p>However, if I remove the podAffinity, it works fine and will overwrite the current node with the new pod from the deployment. What is the correct way to do this to ensure my deployment on my production cluster will always have a pod scheduled on a different node in a different availability zone and also be able to update the existing nodes?</p>

<p>Your goal can be achieved using only <a href="https://docs.openshift.com/container-platform/3.6/admin_guide/scheduling/pod_affinity.html" rel="nofollow noreferrer">PodAntiAffinity</a>.</p>
<p>I have tested this with my <code>GKE</code> test cluster, but it should work similar on <code>Azure</code>.</p>
<h3>Current Issue</h3>
<p>In your current setup, you have set <code>podAntiAffinity</code> with <code>nodeAffinity</code>.</p>
<blockquote>
<p><code>Pod anti-affinity</code> can prevent the scheduler from locating a new pod on the same node as pods with the same labels if the label selector on the new pod matches the label on the current pod.</p>
</blockquote>
<p>In your <code>Deployment</code> setup, new pods will have <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/" rel="nofollow noreferrer">labels</a> like:</p>
<ul>
<li><code>app: auth</code></li>
<li><code>environment: dev</code></li>
<li><code>app-role: api</code></li>
<li><code>tier: backend</code></li>
</ul>
<p><code>PodAntiAffinity</code> was configured to <strong>not allow</strong> deploying new pod, if there is already pod with label: <code>app: auth</code>.</p>
<p><code>NodeAffinity</code> was configured to deploy <strong>only on the node</strong> with label <code>environment: dev</code>.</p>
<p>To Sum up, your error:</p>
<pre><code>0/4 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match node selector.
</code></pre>
<blockquote>
<p>1 node(s) didn't match pod affinity/anti-affinity</p>
</blockquote>
<p>your setup allows to deploy only on the node with label <code>environment: dev</code> and only one pod with label <code>app: auth</code>.</p>
<p>As you mention</p>
<blockquote>
<p>if I already have pods scheduled on a node, then when I do a deployment it will not overwrite the pods that already exist.</p>
</blockquote>
<p><code>PodAntiAffinity</code> behavior worked and didn't allow to deploy new pod with label <code>app: auth</code> as there was already one.</p>
<blockquote>
<p>3 node(s) didn't match node selector.</p>
</blockquote>
<p><code>NodeAffinity</code> allows to deploy pods only on the node with label <code>environment: dev</code>. Other nodes have probably labels like <code>environment: system</code>, <code>environment: uat</code>, <code>environment: qa</code> which didn't match <code>environment: dev</code> label thus didn't match <code>node selector</code>.</p>
<h3>Solution</h3>
<p>Easiest way is to remove <code>NodeAffinity</code>.</p>
<p>While <code>TolpologyKey</code> is set to <code>kubernetes.io/hostname</code> in <code>PodAntiAffinity</code> it's enough.</p>
<blockquote>
<p>The topologyKey uses the default label attached to a node to dynamically filter on the name of the node.</p>
</blockquote>
<p>For more information, please check <a href="https://thenewstack.io/strategies-for-kubernetes-pod-placement-and-scheduling/" rel="nofollow noreferrer">this article</a>.</p>
<p>If you will describe your <code>nodes</code> and <code>grep</code> them with <code>kubernetes.io/hostname</code> you will get unique value:</p>
<pre><code>$ kubectl describe node | grep kubernetes.io/hostname
                    kubernetes.io/hostname=gke-affinity-default-pool-27d6eabd-vhss
                    kubernetes.io/hostname=gke-affinity-default-pool-5014ecf7-5tkh
                    kubernetes.io/hostname=gke-affinity-default-pool-c2afcc97-clg9
</code></pre>
<h3>Tests</h3>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth
  labels:
    app: auth
    environment: dev
    app-role: api
    tier: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: auth
  template:
    metadata:
      labels:
        app: auth
        environment: dev
        app-role: api
        tier: backend
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - auth
            topologyKey: kubernetes.io/hostname
      containers:
        - name: nginx
          image: nginx
          imagePullPolicy: Always
          ports:
            - containerPort: 80
</code></pre>
<p>After deploying this YAML.</p>
<pre><code>$ kubectl get po -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP         NODE                                      NOMINATED NODE   READINESS GATES
auth-7fccf5f7b8-4dkc4   1/1     Running   0          9s    10.0.1.9   gke-affinity-default-pool-c2afcc97-clg9   &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-5qgt4   1/1     Running   0          8s    10.0.2.6   gke-affinity-default-pool-5014ecf7-5tkh   &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-bdmtw   1/1     Running   0          8s    10.0.0.9   gke-affinity-default-pool-27d6eabd-vhss   &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>If you would increase replicas to 7, no more pods will be deployed. All new pods will stuck in <code>Pending</code> state, as <code>antiPodAffinity</code> worked (each node already have pod with label <code>app: dev</code>).</p>
<pre><code>$ kubectl get po -o wide
NAME                    READY   STATUS    RESTARTS   AGE    IP         NODE                                      NOMINATED NODE   READINESS GATES
auth-7fccf5f7b8-4299k   0/1     Pending   0          79s    &lt;none&gt;     &lt;none&gt;                                    &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-4dkc4   1/1     Running   0          2m1s   10.0.1.9   gke-affinity-default-pool-c2afcc97-clg9   &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-556h5   0/1     Pending   0          78s    &lt;none&gt;     &lt;none&gt;                                    &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-5qgt4   1/1     Running   0          2m     10.0.2.6   gke-affinity-default-pool-5014ecf7-5tkh   &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-bdmtw   1/1     Running   0          2m     10.0.0.9   gke-affinity-default-pool-27d6eabd-vhss   &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-q4s2c   0/1     Pending   0          79s    &lt;none&gt;     &lt;none&gt;                                    &lt;none&gt;           &lt;none&gt;
auth-7fccf5f7b8-twb9j   0/1     Pending   0          79s    &lt;none&gt;     &lt;none&gt;                                    &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>Similar solution was described in <a href="https://www.alibabacloud.com/blog/high-availability-deployment-of-pods-on-multi-zone-worker-nodes_595085" rel="nofollow noreferrer">High-Availability Deployment of Pods on Multi-Zone Worker Nodes</a> blog.</p>
