<p>I am having a cluster on EKS with cluster autoscaler enabled. Lets assume there are 3 nodes node-1,node-2,node3. The nodes can have maximum of 10 pods each. Now when the 31st pod comes into picture the CA will launch a new node and schedule the pod on that. Now maybe lets say the 4 pods from node 2 are not required and they go down. Now according to the requirement if a new pod is launched the scheduler places the new pod on the 4th node (launched by the CA) and not on the second node. Also I want that going down further if the pods are removed from the nodes then the new pods should come into the already existing node and not in a new node put up by CA. I tried updating the EKS default scheduler config file using a scheduler plugin but am unable to do so.</p>
<p>I think we can create a second scheduler but I am not aware of the process properly. Any workaround or suggestions will help a lot.</p>
<p>This is the command:
<strong>&quot;kube-scheduler --config custom.config&quot;</strong>
and this is the error
<strong>&quot;attempting to acquire leader lease kube-system/kube-scheduler...&quot;</strong></p>
<p>This is my custom.config file</p>
<pre><code>apiVersion: kubescheduler.config.k8s.io/v1beta1
clientConnection:
   kubeconfig: /etc/kubernetes/scheduler.conf
kind: KubeSchedulerConfiguration
percentageOfNodesToScore: 100
profiles:
- schedulerName: kube-scheduler-new
  plugins:
   score:
  disabled:
    - name: '*'
  enabled:
    - name: NodeResourcesMostAllocated
</code></pre>

<p><strong>How to manage pods scheduling?</strong></p>
<p>Custom scheduler is, of course one way to go if you have some specific use case but if you just want to have some particular node that you want to schedule the pod into to Kubernetes provides an options to do so.</p>
<p>Scheduling algorithm selection can be broken into two parts:</p>
<ul>
<li>Filtering the list of all nodes to obtain a list of acceptable nodes the pod can be scheduled to.</li>
<li>Prioritizing the acceptable nodes and choosing the best one. If multiple nodes have the highest score, round-robin is used to ensure pods are deployed across all of them evenly</li>
</ul>
<p>Kubernetes works great if you let scheduler decides which nodes the pod should go and it comes with tools that will give scheduler hints:</p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/" rel="nofollow noreferrer">Taints and tolerations</a> can be used to repel some certain pods from the node. This is very useful if you want to partition your cluster and allow some certain people schedule into some specific nodes. Can be also used when you have some hardware nodes and some pod require it (like in your question where you want a pod to be schedule on node 2). They come with 3 effects:</li>
</ul>
<ol>
<li><code>NoSchedule</code> which means there will be no scheduling</li>
<li><code>PreferNoSchedule</code> which means scheduler will try to avoid scheduling</li>
<li><code>NoExecute</code> also affects scheduling and affects pods already running on the node. IF you add this taint to node, pods that are running on the node and don't tolerate that will be evicted.</li>
</ol>
<ul>
<li><p><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity" rel="nofollow noreferrer">Node affinity</a> on the other side can be used to attract some certain pods into specific nodes. Similar to tains node affinity does give me some options for fine tuning your scheduling preferences:</p>
<ol>
<li><code>requiredDuringSchedulingIgnoredDuringExecution</code> which can be used as hard requirement and tell scheduler that rules must be met for pod to be scheduled onto node.</li>
<li><code>preferredDuringSchedulingIgnoredDuringExecution</code> which can be used as soft requirement and tell scheduler to try to enforce it but it does not have to be guaranteed</li>
</ol>
</li>
<li><p><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="nofollow noreferrer">PodAffinity</a> can be used if you for example want your front-end pods to run on the same node as your database pod. This can be in similar way described as hard or soft requirement respectively.</p>
</li>
<li><p><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="nofollow noreferrer">podAntiAffinity</a> can be used if you wish not have some certain pod to be running with each other.</p>
</li>
</ul>
