<p>We have 4 nodes EKS cluster. We have some pods (part of Daemonset) in pending status as the Node is full and there is no capacity in the node to run the pod. The question is do we need to manually reshuffle the workloads to make Daemonset pods running in this situation or is there any configuration to overcome this issue in an automated fashion?</p>
<p>Note: we have also installed Cluster Autoscaler which works perfectly for deployments.</p>
<p>Thank you in advance.</p>

Bhaumik Patel
<p>Kubernetes has pod priorities and preemption for this specific purpose.</p>
<blockquote>
<p>Pods can have priority. Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible.<br />
ref: <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/</a></p>
</blockquote>
<p>If EKS does not have priority classes pre-configured, you can create one yourself. For example, the one from the docs which is a preempting one:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: &quot;This priority class should be used for XYZ service pods only.&quot;
</code></pre>
<p>Then you use that class on your daemon set</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: high-priority # this is important
</code></pre>
<p>Note this is just a small example copied from linked docs, you should read the docs carefully and perhaps also review how this would interact with pod disruption budgets.</p>
<p>Also note, that this may cause disruption to other deployments, depending on various factors such as the Update Strategy. So, be careful.</p>
