<p>We would like to pack as many pods into each nodes in our cluster as much as possible do decrease the amount of nodes we have on some of our environments. I saw <a href="https://github.com/kubernetes-sigs/descheduler" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/descheduler</a> HighNodeUtilization strategy which seems to fit the bill for what we need. However, it seems the cluster needs to have the scoring strategy <strong>MostAllocated</strong> to work with this.</p>
<p>I believe that the kube-scheduler in EKS in inaccessible to be configured. How do I then configure the MostAllocated scoring strategy?</p>
<p>Better yet, how do I configure this automated packing of pods in as little nodes as possible in a cluster without the use of Descheduler?</p>
<p>Tried deploying the descheduler as is without the MostAllocated scoring strategy configured. Obviously did not provide the results expected.</p>
<p>Many of my digging online led to having to create a custom-scheduler, but I have found little/unclear resources to be able to do so.</p>

<p>Eks does not provide the ability to override the default scheduler configuration, which means that actually configuring the <code>default-scheduler</code> profile with the <code>MostAllocated</code> scoring strategy is not an option.  However, you may run your own scheduler <em>alongside</em> the default scheduler, and <a href="https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/" rel="nofollow noreferrer">this one may be configured how you like</a>.  Once you create a custom scheduler, you can override <em>that</em> scheduler's configuration with the <code>MostAllocated</code> scoring strategy and then instruct your workloads to use that scheduler.</p>
<p>In order to run multiple schedulers, you have to set up several Kubernetes Objects.  These objects are documented in the guide linked above:</p>
<ul>
<li>ServiceAccount</li>
<li>ClusterRoleBinding x2</li>
<li>RoleBinding</li>
<li>ConfigMap</li>
<li>Deployment</li>
</ul>
<p>The deployment will use the standard <code>kube-scheduler</code> image provided by Google, <a href="https://www.youtube.com/watch?v=IYcL0Un1io0" rel="nofollow noreferrer">unless you'd like to create your own</a>.  I wouldn't recommend it.</p>
<h3>Major Note: Ensure your version of the kube-scheduler is the same version as the control plane.  This will not work otherwise.</h3>
<p>In addition, ensure that your version of the <code>kube-scheduler</code> is compatible with the version of the configuration objects that you use to configure the scheduler profile.  <code>v1beta2</code> is safe for <code>v1.22.x</code> -&gt; <code>v1.24.x</code> but only <code>v1beta3</code> or <code>v1</code> is safe for <code>v.1.25+</code>.</p>
<p>For example, here's a working version of a deployment manifest and config map that are used to create a custom scheduler compatible with <code>k8s</code> <code>v.1.22.x</code>.  Note you'll still have to create the other objects for this to work:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-scheduler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      name: custom-scheduler
  template:
    metadata:
      labels:
        component: scheduler
        name: custom-scheduler
        tier: control-plane
    spec:
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        - --config=/etc/kubernetes/custom-scheduler/custom-scheduler-config.yaml
        env: []
        image: registry.k8s.io/kube-scheduler:v1.22.16
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        name: custom-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        volumeMounts:
        - mountPath: /etc/kubernetes/custom-scheduler
          name: custom-scheduler-config
      serviceAccountName: custom-scheduler
      volumes:
      - configMap:
          name: custom-scheduler-config
        name: custom-scheduler-config
</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ConfigMap
data:
  custom-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1beta2
    kind: KubeSchedulerConfiguration
    leaderElection:
        leaderElect: false
    profiles:
        - pluginConfig:
            - args:
                apiVersion: kubescheduler.config.k8s.io/v1beta2
                kind: NodeResourcesFitArgs
                scoringStrategy:
                    resources:
                        - name: cpu
                          weight: 1
                        - name: memory
                          weight: 1
                    type: MostAllocated
              name: NodeResourcesFit
          plugins:
            score:
                enabled:
                    - name: NodeResourcesFit
                      weight: 1
          schedulerName: custom-scheduler
metadata:
  name: custom-scheduler-config
  namespace: kube-system
</code></pre>
