<p>I have series of deployments on k8s that deploys same server binary but different in spec like memory limit, etc. each deployments only 1 pod to run, and want to schedlule pods for these deployments so that each VMs are only scheduled single pod.</p>
<p>because some deployments require larger memory than others, we combine nodeAffinity and podAntiAffinity to satisfy following requirement.</p>
<ul>
<li>every type of instances will be assigned only single pod which has label <code>pod_group_affinity</code> == <code>per_node</code></li>
<li>each pod that has different memory requirement correcty assigned to the node that has corresponding memory size</li>
</ul>
<p>below is my configuration.</p>
<pre><code>    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - ${INSTANCE_TYPE}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod_group_affinity
                operator: In
                values:
                - per_node
            topologyKey: &quot;kubernetes.io/hostname&quot;
</code></pre>
<p>I first tried with local minikube as single VM (with giving proper node.kubernetes.io/instance-type value), but pods of all deployments are scheduled. it should not happen because all pods have pod_group_affinity == per_node label. even I remove nodeAffinity part, still podAntiAffinity does not seem to work.</p>
<p>am I missing something? for example, podAntiAffinity does not work for inter pods from multiple deployment? but as long as I read the article <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#more-practical-use-cases" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#more-practical-use-cases</a> , podAntiAffinity can refer the label of pods from another deployment.</p>
<p>regards,</p>
<p>EDIT: I added more information to investigate the reason, due to the suggestion of @confused genius.</p>
<ul>
<li><code>kubectl get nodes --show-labes</code></li>
</ul>
<pre><code>$ kubectl get nodes --show-labels
NAME       STATUS   ROLES                  AGE     VERSION   LABELS
minikube   Ready    control-plane,master   4d19h   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=minikube,kubernetes.io/os=linux,minikube.k8s.io/commit=0a0ad764652082477c00d51d2475284b5d39ceed,minikube.k8s.io/name=minikube,minikube.k8s.io/updated_at=2022_01_19T12_04_46_0700,minikube.k8s.io/version=v1.23.2,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=,node.kubernetes.io/instance-type=Standard_B2ms,topology.hostpath.csi/node=minikube
</code></pre>
<p>I manually set <code>node.kubernetes.io/instance-type=Standard_B2ms</code> to emurate real environment which each VM has label of corresponding instance type.</p>
<ul>
<li>template part of deployment
due to NDA of the work, I cannot put real version of deployment, but it is almost like below.</li>
</ul>
<pre><code>spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${NODE_NAME}
      pod_group_affinity: per_node
  template:
    metadata:
      labels:
        app: ${NODE_NAME}
        team: backend
        release: stable
        environment: ${ENV_LABEL}
        pod_group_affinity: per_node
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - ${INSTANCE_TYPE}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod_group_affinity
                operator: In
                values:
                - per_node
            topologyKey: &quot;kubernetes.io/hostname&quot;                
      initContainers:
      - name: registration
        image: ${REGISTRATION_CONTAINER_IMAGE}
        env:
          - name: PASSWORD
            value: &quot;${PASSWORD}&quot;
        imagePullPolicy: IfNotPresent
        command: [&quot;/bin/bash&quot;, &quot;-c&quot;, &quot;/workdir/run.sh&quot;]
        volumeMounts:
        - name: node-config
          mountPath: /workdir/run.sh
          subPath: run.sh
        resources:
          requests:
            memory: &quot;${MEMORY_REQUESTS}&quot;
            cpu: &quot;${CPU_REQUESTS}&quot;
          limits:
            memory: &quot;${MEMORY_LIMITS}&quot;
      containers:
      - name: web
        image: ${WEB_CONTAINER_IMAGE}
        env:
          - name: USERNAME
            value: &quot;${USERNAME}&quot;
          - name: PASSWORD
            value: &quot;${PASSWORD}&quot;
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: rest
          protocol: TCP
      - name: cordapp
        image: ${NODE_CONTAINER_IMAGE}
        env:
          - name: QUEUE_REGION
            valueFrom:
              configMapKeyRef:
                name: ${NODE_NAME}-config
                key: QUEUE_REGION
        imagePullPolicy: IfNotPresent
        command: [&quot;/bin/bash&quot;, &quot;-c&quot;, &quot;/workdir/run.sh&quot;]
        lifecycle:
          postStart:
            exec:
              command: [&quot;/workdir/lifecycle.sh&quot;, &quot;startup&quot;, &quot;${GRACE_PERIOD_SECONDS}&quot;]
          preStop:
            exec:
              command: [&quot;/workdir/lifecycle.sh&quot;, &quot;shutdown&quot;, &quot;${GRACE_PERIOD_SECONDS}&quot;]
        ports:
        - containerPort: 10000
          name: rpc
        volumeMounts:
        resources:
          requests:
            memory: &quot;${MEMORY_REQUESTS}&quot;
            cpu: &quot;${CPU_REQUESTS}&quot;
          limits:
            memory: &quot;${MEMORY_LIMITS}&quot;
      terminationGracePeriodSeconds: ${GRACE_PERIOD_SECONDS}
      volumes:
      - name: ${NODE_NAME}-config
        configMap:
          name: ${NODE_NAME}-config
</code></pre>

<p>I could solve the problem.</p>
<p>The reason that above settings do not work well is I put each deployment in different namespace, as described in <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity</a></p>
<p>Final yaml will look like:</p>
<pre class="lang-yaml prettyprint-override"><code>      podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod_group_affinity
                operator: In
                values:
                - per_node
            topologyKey: &quot;kubernetes.io/hostname&quot;
            # this setting is important
            namespaces: [&quot;node1&quot;, &quot;node2&quot;, &quot;node3&quot;]
</code></pre>
