<p>I am running Kubernetes with Rancher, and I am seeing weird behavior with the kube-scheduler. After adding a third node, I expect to see pods start to get scheduled &amp; assigned to it. However, the kube-scheduler scores this new third node <code>node3</code> with the lowest score, even though it has almost no pods running in it, and I expect it to receive the highest score.</p>

<p>Here are the logs from the Kube-scheduler:</p>

<pre><code>scheduling_queue.go:815] About to try and schedule pod namespace1/pod1
scheduler.go:456] Attempting to schedule pod: namespace1/pod1
predicates.go:824] Schedule Pod namespace1/pod1 on Node node1 is allowed, Node is running only 94 out of 110 Pods.
predicates.go:1370] Schedule Pod namespace1/pod1 on Node node1 is allowed, existing pods anti-affinity terms satisfied.
predicates.go:824] Schedule Pod namespace1/pod1 on Node node3 is allowed, Node is running only 4 out of 110 Pods.
predicates.go:1370] Schedule Pod namespace1/pod1 on Node node3 is allowed, existing pods anti-affinity terms satisfied.
predicates.go:824] Schedule Pod namespace1/pod1 on Node node2 is allowed, Node is running only 95 out of 110 Pods.
predicates.go:1370] Schedule Pod namespace1/pod1 on Node node2 is allowed, existing pods anti-affinity terms satisfied.
resource_allocation.go:78] pod1 -&gt; node1: BalancedResourceAllocation, capacity 56000 millicores 270255251456 memory bytes, total request 40230 millicores 122473676800 memory bytes, score 7
resource_allocation.go:78] pod1 -&gt; node1: LeastResourceAllocation, capacity 56000 millicores 270255251456 memory bytes, total request 40230 millicores 122473676800 memory bytes, score 3
resource_allocation.go:78] pod1 -&gt; node3: BalancedResourceAllocation, capacity 56000 millicores 270255251456 memory bytes, total request 800 millicores 807403520 memory bytes, score 9
resource_allocation.go:78] pod1 -&gt; node3: LeastResourceAllocation, capacity 56000 millicores 270255251456 memory bytes, total request 800 millicores 807403520 memory bytes, score 9
resource_allocation.go:78] pod1 -&gt; node2: BalancedResourceAllocation, capacity 56000 millicores 270255247360 memory bytes, total request 43450 millicores 133693440000 memory bytes, score 7
resource_allocation.go:78] pod1 -&gt; node2: LeastResourceAllocation, capacity 56000 millicores 270255247360 memory bytes, total request 43450 millicores 133693440000 memory bytes, score 3
generic_scheduler.go:748] pod1_namespace1 -&gt; node1: TaintTolerationPriority, Score: (10)
generic_scheduler.go:748] pod1_namespace1 -&gt; node3: TaintTolerationPriority, Score: (10)
generic_scheduler.go:748] pod1_namespace1 -&gt; node2: TaintTolerationPriority, Score: (10)
selector_spreading.go:146] pod1 -&gt; node1: SelectorSpreadPriority, Score: (10)
selector_spreading.go:146] pod1 -&gt; node3: SelectorSpreadPriority, Score: (10)
selector_spreading.go:146] pod1 -&gt; node2: SelectorSpreadPriority, Score: (10)
generic_scheduler.go:748] pod1_namespace1 -&gt; node1: SelectorSpreadPriority, Score: (10)
generic_scheduler.go:748] pod1_namespace1 -&gt; node3: SelectorSpreadPriority, Score: (10)
generic_scheduler.go:748] pod1_namespace1 -&gt; node2: SelectorSpreadPriority, Score: (10)
generic_scheduler.go:748] pod1_namespace1 -&gt; node1: NodeAffinityPriority, Score: (0)
generic_scheduler.go:748] pod1_namespace1 -&gt; node3: NodeAffinityPriority, Score: (0)
generic_scheduler.go:748] pod1_namespace1 -&gt; node2: NodeAffinityPriority, Score: (0)
 interpod_affinity.go:232] pod1 -&gt; node1: InterPodAffinityPriority, Score: (0)
 interpod_affinity.go:232] pod1 -&gt; node3: InterPodAffinityPriority, Score: (0)
interpod_affinity.go:232] pod1 -&gt; node2: InterPodAffinityPriority, Score: (10)
generic_scheduler.go:803] Host node1 =&gt; Score 100040
generic_scheduler.go:803] Host node3 =&gt; Score 100038
generic_scheduler.go:803] Host node2 =&gt; Score 100050
scheduler_binder.go:256] AssumePodVolumes for pod "namespace1/pod1", node "node2"
scheduler_binder.go:266] AssumePodVolumes for pod "namespace1/pod1", node "node2": all PVCs bound and nothing to do
factory.go:727] Attempting to bind pod1 to node2
</code></pre>

<p>I can tell from the logs that your pod will always be scheduled on <code>node2</code> because it seems like you have some sort of <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="nofollow noreferrer">PodAffinity</a> that scores an additional <code>10</code>. Making it go to <code>50</code>. </p>

<p>What's kind of odd is that I'm scoring <code>48</code> for node3 but it seems like <code>-10</code> is being stuck there somewhere (totaling <code>38</code>). Perhaps because of the affinity, or some entry not being shown in the logs, or plain simply a bug in the way the scheduler is doing the calculation. You'll probably have to dig deep into the <a href="https://github.com/kubernetes/kubernetes/tree/master/pkg/scheduler" rel="nofollow noreferrer">kube-scheduler code</a> if you'd like to find out more.</p>

<p>This is what I have:</p>

<pre><code>node1 7 + 3 + 10 + 10 + 10 = 40
node2 7 + 3 + 10 + 10 + 10 + 10 = 50
node3 9 + 9 + 10 + 10 + 10 = 48
</code></pre>
