<p>I am working on migrating my applications to Kubernetes. I am using EKS.</p>
<p>I want to distribute my pods to different nodes, to avoid having a single point of failure.
I read about <code>pod-affinity</code> and <code>anti-affinity</code> and <code>required</code> and <code>preferred</code> mode.</p>
<p><a href="https://stackoverflow.com/a/49900137/3333052">This answer</a> gives a very nice way to accomplish this.</p>
<p>But my doubt is, let's say if I have 3 nodes, of which 2 are already full(resource-wise). If I use <code>requiredDuringSchedulingIgnoredDuringExecution</code>, k8s will spin up new nodes and will distribute the pods to each node. And if I use <code>preferredDuringSchedulingIgnoredDuringExecution</code>, it will check for preferred-nodes, and not finding different nodes, will deploy all pods on the third node only. In which case, it will again become a single point of failure.</p>
<p>How do I solve this condition?</p>
<p>One way I can think of is to have an over-provisioned cluster, so that there are always some extra nodes.</p>
<p>The second way, I am not sure how to do this, but I think there should be a way of using both <code>requiredDuringSchedulingIgnoredDuringExecution</code> and <code>preferredDuringSchedulingIgnoredDuringExecution</code>.</p>
<p>Can anyone help me with this? Am I missing something? How do people work with this condition?</p>
<p>I am new to Kubernetes, so feel free to correct me if I am wrong or missing something.</p>
<p>Thanks in advance</p>
<p><strong>Note:</strong></p>
<p>I don't have a problem running a few similar pods on the same node, just don't want all of the pods to be running on the same node, just because there was only one node available to deploy.</p>

kadamb
<p>I see you are trying to make sure that k8s will never schedule all pod replicas on the same node.</p>

<p>It's not possible to create hard requrement like this for kubernetes scheduler.</p>

<p>Scheduler will try its best to schedule your application as evenly as possible but in situation when you have 2 nodes without spare resources and 1 node where all pod replicas would be scheduled, k8s can do one of the folowing actions (depending on configuration):</p>

<ol>
<li>schedule your pods on one node (best effort/default)</li>
<li>run one pod and not schedule rest of the pods at all (<code>antiaffnity</code> + <code>requiredDuringSchedulingIgnoredDuringExecution</code>)</li>
<li>create new nodes for pods if needed (<code>antiaffnity</code> + <code>requiredDuringSchedulingIgnoredDuringExecution</code> + <code>cluster autoscaler</code>)</li>
<li>start deleting pods from nodes to free resources for high-priority pods (<a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/" rel="nofollow noreferrer"><code>priority based preemption</code></a>) and reschedule preempted pods if possible</li>
</ol>

<p>Also read this <a href="https://medium.com/expedia-group-tech/how-to-keep-your-kubernetes-deployments-balanced-across-multiple-zones-dfe719847b41" rel="nofollow noreferrer">article</a> to get better understanding on how scheduler makes its decisions.</p>

<p>You can also use <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/" rel="nofollow noreferrer">PodDisruptionBudget</a> to tell kubernetes to make sure a specified replicas are always working, remember that although:</p>

<blockquote>
  <p>A disruption budget does not truly guarantee that the specified number/percentage of pods will always be up.</p>
</blockquote>

<p>kubernetes will take it under consideration when making scheduling decisions.</p>
