<p>So I'm not sure if I'm using this feature incorrectly or it's a limitation of &quot;anti-affinity&quot; but I'm at a loss. I have a batch of jobs that I want to run with each one having their own node. Should be relatively simple, add the anti-affinity to only run the pod where the hostname + label does not exist. Despite this I still have multiple pods on the same node.</p>
<p>My best guess right now is that, because I create all the jobs at once with a <code>kubectl apply -f ./folder</code> command, the scheduler doesn't count pods on a node in the &quot;container creating&quot; state as a trigger for the anti-affinity rule and schedules another onto the node.</p>
<p>Each job needs a slightly different command line so I can't use just one job file with the parallel spec until 1.22 comes out with job indexing.</p>
<p>Below is the job yaml in case there's something I'm missing.</p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
 name: testjob-$SHARD
spec:
 backoffLimit: 1
 template:
   metadata:
     labels:
       run: testjob
   spec:
     affinity:
       podAntiAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
         - labelSelector:
             matchExpressions:
             - key: run
               operator: In
               values:
               - testjob
           topologyKey: &quot;kubernetes.io/hostname&quot;
     containers:
     - name: testjob
       imagePullPolicy: Always
       image: image
       resources:
         requests:
           memory: &quot;3072Mi&quot;
         limits:
           memory: &quot;4Gi&quot;
       command: [&quot;./foo&quot;]
       securityContext:
         privileged: true
     restartPolicy: OnFailure
</code></pre>

<p>So as a shot in the dark I upgraded my cluster from 1.20 to 1.21 and it magically fixed itself. So it's possible this was just a 1.20 problem. I have a issue on the kubernetes github to look into it but all I can say is that if you're on 1.20 and having trouble with pod level anti-affinity maybe try upgrading.</p>
