<p>I have a terraform-managed EKS cluster. It used to have 2 nodes on it. I doubled the number of nodes (4).</p>
<p>I have a kubernetes_deployment resource that automatically deploys a fixed number of pods to the cluster. It was set to 20 when I had 2 nodes, and seemed evenly distributed with 10 each. I doubled that number to 40.</p>
<p>All of the new pods for the kubernetes deployment are being scheduled on the first 2 (original) nodes. Now the two original nodes have 20 pods each, while the 2 new nodes have 0 pods. The new nodes are up and ready to go, but I cannot get kubernetes to schedule the new pods on those new nodes.</p>
<p>I am unsure where to even begin searching, as I am fairly new to k8s and ops in general.</p>
<p>A few beginner questions that may be related:</p>
<ol>
<li><p>I'm reading about pod affinity, and it seems like I could tell k8s to have a pod ANTI affinity with itself within a deployment. However, I am having trouble setting up the anti-affinity rules. I see that the <a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/deployment#preferred_during_scheduling_ignored_during_execution" rel="nofollow noreferrer">kubernetes_deployment</a> resource has a scheduling argument, but I can't seem to get the syntax right.</p>
</li>
<li><p>Naively it seems that the issue may be that the deployment somehow isn't aware of the new nodes. If that is the case, how could I reboot the entire deployment (without taking down the already-running pods)?</p>
</li>
<li><p>Is there a cluster level scheduler that I need to set? I was under the impression that the default does round robin, which doesn't seem to be happening at the node level.</p>
</li>
</ol>
<p>EDIT:
The EKS terraform module <a href="https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/11.0.0/submodules/node_groups" rel="nofollow noreferrer">node_groups submodule</a> has fields for desired/min/max_capacity. To increase my worker nodes, I just increased those numbers. The change is reflected in the aws eks console.</p>

<p>Check a couple of things:</p>
<ol>
<li>Do your nodes show up correctly in the output of <code>kubectl get nodes -o wide</code> and do they have a state of ready?</li>
<li>Instead of pod affinity look into <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/" rel="nofollow noreferrer">pod topology spread constraints</a>. Anti affinity will not work with multiple pods.</li>
</ol>
